[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:10:36.716266 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:43933,127.23.244.61:37851,127.23.244.60:43347
I20240501 08:10:36.717909 24528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:36.719012 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:36.731070 24534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:36.731289 24533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:36.731860 24536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.732775 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:36.733520 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:36.733680 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:36.733760 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551036733739 us; error 0 us; skew 500 ppm
I20240501 08:10:36.734118 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:36.736670 24528 webserver.cc:460] Webserver started at http://127.23.244.62:45867/ using document root <none> and password file <none>
I20240501 08:10:36.737385 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:36.737483 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:36.737810 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:36.741101 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.746510 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:36.749677 24541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:36.751055 24528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:36.751233 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.751427 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:36.787604 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:36.788501 24528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:36.788807 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:36.809154 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:43933
I20240501 08:10:36.809181 24592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:43933 every 8 connection(s)
I20240501 08:10:36.811401 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:36.814465 24595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.814536 24593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:10:36.815372 24596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.815840 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:36.816819 24598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.817257 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:36.817348 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:36.817423 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551036817385 us; error 0 us; skew 500 ppm
I20240501 08:10:36.817625 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:36.819104 24528 webserver.cc:460] Webserver started at http://127.23.244.61:37833/ using document root <none> and password file <none>
I20240501 08:10:36.819379 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:36.819490 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:36.819634 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:36.820525 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root/instance:
uuid: "b77b8451c1de4472a1fdbb549713f6a7"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.823990 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:36.826128 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:36.826467 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:36.826558 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
uuid: "b77b8451c1de4472a1fdbb549713f6a7"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.826694 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:36.829636 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:36.846881 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:36.848117 24542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.244.61:37851: connect: Connection refused (error 111)
I20240501 08:10:36.848570 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:36.849484 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 08:10:36.849979 24593 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.61:37851: Network error: Client connection negotiation failed: client connection to 127.23.244.61:37851: connect: Connection refused (error 111)
I20240501 08:10:36.865823 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.61:37851
I20240501 08:10:36.865859 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.61:37851 every 8 connection(s)
I20240501 08:10:36.867208 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:36.867357 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:10:36.870621 24660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.870692 24658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:36.871459 24661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.877408 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:36.878656 24663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:36.879092 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:36.879165 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:36.879240 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551036879203 us; error 0 us; skew 500 ppm
I20240501 08:10:36.879407 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:36.881114 24528 webserver.cc:460] Webserver started at http://127.23.244.60:45451/ using document root <none> and password file <none>
I20240501 08:10:36.881400 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:36.881531 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:36.881700 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:36.882519 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root/instance:
uuid: "a3edac21ac7f4e44b24d86df35574f59"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.882609 24658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:36.885562 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 08:10:36.887953 24670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:36.888401 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:36.888535 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
uuid: "a3edac21ac7f4e44b24d86df35574f59"
format_stamp: "Formatted at 2024-05-01 08:10:36 on dist-test-slave-fck4"
I20240501 08:10:36.888701 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:36.890458 24658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:36.892645 24658 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.60:43347: Network error: Client connection negotiation failed: client connection to 127.23.244.60:43347: connect: Connection refused (error 111)
I20240501 08:10:36.899329 24593 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } attempt: 1
I20240501 08:10:36.899935 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:36.901038 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:36.906436 24593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:36.908643 24593 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.60:43347: Network error: Client connection negotiation failed: client connection to 127.23.244.60:43347: connect: Connection refused (error 111)
I20240501 08:10:36.918987 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.60:43347
I20240501 08:10:36.919054 24722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.60:43347 every 8 connection(s)
I20240501 08:10:36.920049 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:10:36.920516 24723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:36.922866 24723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:36.929605 24723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:36.934801 24723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:36.944798 24658 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } attempt: 1
I20240501 08:10:36.945142 24723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: Bootstrap starting.
I20240501 08:10:36.949157 24723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:36.950629 24723 log.cc:824] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:36.951890 24593 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } attempt: 1
I20240501 08:10:36.952610 24658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: Bootstrap starting.
I20240501 08:10:36.953743 24723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: No bootstrap required, opened a new log
I20240501 08:10:36.954340 24658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:36.957038 24658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: No bootstrap required, opened a new log
I20240501 08:10:36.962283 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: Bootstrap starting.
I20240501 08:10:36.963928 24658 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.963940 24723 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.964258 24658 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:36.964255 24723 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:36.964412 24658 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77b8451c1de4472a1fdbb549713f6a7, State: Initialized, Role: FOLLOWER
I20240501 08:10:36.964466 24723 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3edac21ac7f4e44b24d86df35574f59, State: Initialized, Role: FOLLOWER
I20240501 08:10:36.964504 24593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:36.965287 24658 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.965287 24723 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.966763 24593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: No bootstrap required, opened a new log
I20240501 08:10:36.967507 24593 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.967716 24593 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:36.967820 24593 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6eaaa2f6084a22a04a4591e3bc7dc6, State: Initialized, Role: FOLLOWER
I20240501 08:10:36.967823 24723 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:36.968104 24593 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:36.969147 24593 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:36.969094 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:36.969643 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:36.973277 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:36.974064 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:36.975410 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:36.975765 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:36.975765 24658 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:36.978066 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 08:10:36.990438 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240501 08:10:36.992913 24746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:36.993045 24746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 08:10:36.993228 24760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:36.993356 24760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:10:36.996176 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:36.996325 24766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:36.996407 24766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:10:36.999346 24731 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:36.999619 24731 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
W20240501 08:10:37.000921 24767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.001637 24731 leader_election.cc:288] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf6eaaa2f6084a22a04a4591e3bc7dc6 (127.23.244.62:43933), a3edac21ac7f4e44b24d86df35574f59 (127.23.244.60:43347)
W20240501 08:10:37.001780 24768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.002557 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b77b8451c1de4472a1fdbb549713f6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" is_pre_election: true
I20240501 08:10:37.003031 24568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77b8451c1de4472a1fdbb549713f6a7 in term 0.
I20240501 08:10:37.002493 24698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b77b8451c1de4472a1fdbb549713f6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3edac21ac7f4e44b24d86df35574f59" is_pre_election: true
I20240501 08:10:37.003815 24698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77b8451c1de4472a1fdbb549713f6a7 in term 0.
I20240501 08:10:37.004456 24606 leader_election.cc:302] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3edac21ac7f4e44b24d86df35574f59, b77b8451c1de4472a1fdbb549713f6a7; no voters:
I20240501 08:10:37.005038 24731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:37.005318 24731 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:37.005443 24731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:37.010430 24731 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
I20240501 08:10:37.011883 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b77b8451c1de4472a1fdbb549713f6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6"
I20240501 08:10:37.012131 24568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:37.016306 24568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77b8451c1de4472a1fdbb549713f6a7 in term 1.
I20240501 08:10:37.017035 24607 leader_election.cc:302] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b77b8451c1de4472a1fdbb549713f6a7, bf6eaaa2f6084a22a04a4591e3bc7dc6; no voters:
I20240501 08:10:37.019730 24698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b77b8451c1de4472a1fdbb549713f6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3edac21ac7f4e44b24d86df35574f59"
I20240501 08:10:37.019973 24698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:37.025064 24698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77b8451c1de4472a1fdbb549713f6a7 in term 1.
I20240501 08:10:37.024921 24731 leader_election.cc:288] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [CANDIDATE]: Term 1 election: Requested vote from peers bf6eaaa2f6084a22a04a4591e3bc7dc6 (127.23.244.62:43933), a3edac21ac7f4e44b24d86df35574f59 (127.23.244.60:43347)
I20240501 08:10:37.026050 24731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:37.026394 24731 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 1 LEADER]: Becoming Leader. State: Replica: b77b8451c1de4472a1fdbb549713f6a7, State: Running, Role: LEADER
I20240501 08:10:37.027374 24731 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } }
W20240501 08:10:37.030746 24770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.032022 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:37.032316 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:37.032415 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:37.032492 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551037032465 us; error 0 us; skew 500 ppm
I20240501 08:10:37.032234 24772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77b8451c1de4472a1fdbb549713f6a7. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.032696 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:37.032716 24772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:37.033603 24774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:37.035926 24774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240501 08:10:37.040971 24547 tablet.cc:2369] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:37.046605 24528 webserver.cc:460] Webserver started at http://127.23.244.1:42679/ using document root <none> and password file <none>
I20240501 08:10:37.046995 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:37.047119 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:37.047328 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:37.048344 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "42d939bbd4dc452e9b39d7bfe422f17e"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.048616 24698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 1 FOLLOWER]: Refusing update from remote peer b77b8451c1de4472a1fdbb549713f6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:37.048610 24568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 1 FOLLOWER]: Refusing update from remote peer b77b8451c1de4472a1fdbb549713f6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:37.049688 24772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:37.049976 24731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:37.055166 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20240501 08:10:37.065532 24782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.066072 24528 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.008s
I20240501 08:10:37.066259 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "42d939bbd4dc452e9b39d7bfe422f17e"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.066454 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:37.067996 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77b8451c1de4472a1fdbb549713f6a7. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.068361 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:37.070802 24776 mvcc.cc:205] Tried to move back new op lower bound from 7022801047737389056 to 7022801047673561088. Current Snapshot: MvccSnapshot[applied={T|T < 7022801047737389056}]
I20240501 08:10:37.071035 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77b8451c1de4472a1fdbb549713f6a7. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.071293 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:37.071830 24772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.072049 24772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:37.073856 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.074136 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:37.075109 24774 catalog_manager.cc:1347] Generated new cluster ID: 080d079aa96f4ad692bb9f077c411802
I20240501 08:10:37.075230 24774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:37.085346 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.085644 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:37.087754 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b77b8451c1de4472a1fdbb549713f6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6eaaa2f6084a22a04a4591e3bc7dc6" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 43933 } } peers { permanent_uuid: "b77b8451c1de4472a1fdbb549713f6a7" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 37851 } } peers { permanent_uuid: "a3edac21ac7f4e44b24d86df35574f59" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 43347 } } }
I20240501 08:10:37.087992 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:37.103597 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:37.104391 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:37.105971 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:37.106968 24774 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:37.109006 24774 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:37.109194 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:37.109336 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.109452 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:37.109551 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.127642 24774 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: Generated new TSK 0
I20240501 08:10:37.136432 24774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:37.150768 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:38031
I20240501 08:10:37.150840 24850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:38031 every 8 connection(s)
I20240501 08:10:37.165787 24851 heartbeater.cc:347] Connected to a master server at 127.23.244.60:43347
I20240501 08:10:37.166146 24851 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.167768 24851 heartbeater.cc:511] Master 127.23.244.60:43347 requested a full tablet report, sending...
I20240501 08:10:37.171139 24688 ts_manager.cc:194] Registered new tserver with Master: 42d939bbd4dc452e9b39d7bfe422f17e (127.23.244.1:38031)
I20240501 08:10:37.179517 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:37.180533 24854 heartbeater.cc:347] Connected to a master server at 127.23.244.61:37851
I20240501 08:10:37.180717 24854 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.180775 24852 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43933
I20240501 08:10:37.180924 24852 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.181587 24854 heartbeater.cc:511] Master 127.23.244.61:37851 requested a full tablet report, sending...
I20240501 08:10:37.181659 24852 heartbeater.cc:511] Master 127.23.244.62:43933 requested a full tablet report, sending...
I20240501 08:10:37.183075 24623 ts_manager.cc:194] Registered new tserver with Master: 42d939bbd4dc452e9b39d7bfe422f17e (127.23.244.1:38031)
I20240501 08:10:37.183388 24558 ts_manager.cc:194] Registered new tserver with Master: 42d939bbd4dc452e9b39d7bfe422f17e (127.23.244.1:38031)
W20240501 08:10:37.185897 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.186185 24623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53258
W20240501 08:10:37.187243 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.189993 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:37.190661 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.191064 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:37.191133 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:37.191210 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551037191171 us; error 0 us; skew 500 ppm
I20240501 08:10:37.191373 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:37.192904 24528 webserver.cc:460] Webserver started at http://127.23.244.2:36895/ using document root <none> and password file <none>
I20240501 08:10:37.193140 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:37.193244 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:37.193377 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:37.194643 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "cae247bed8d5431d9691590244e260a6"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.197858 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:10:37.199787 24865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.200204 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:10:37.200323 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "cae247bed8d5431d9691590244e260a6"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.200479 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:37.234859 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:37.236091 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:37.236927 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:37.238097 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:37.238186 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.238312 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:37.238379 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.259006 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:38703
I20240501 08:10:37.259016 24927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:38703 every 8 connection(s)
I20240501 08:10:37.274621 24929 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43933
I20240501 08:10:37.274850 24929 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.275674 24929 heartbeater.cc:511] Master 127.23.244.62:43933 requested a full tablet report, sending...
I20240501 08:10:37.277135 24558 ts_manager.cc:194] Registered new tserver with Master: cae247bed8d5431d9691590244e260a6 (127.23.244.2:38703)
I20240501 08:10:37.279997 24928 heartbeater.cc:347] Connected to a master server at 127.23.244.60:43347
I20240501 08:10:37.280167 24928 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.280664 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:37.280794 24928 heartbeater.cc:511] Master 127.23.244.60:43347 requested a full tablet report, sending...
I20240501 08:10:37.281941 24688 ts_manager.cc:194] Registered new tserver with Master: cae247bed8d5431d9691590244e260a6 (127.23.244.2:38703)
W20240501 08:10:37.285360 24934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:37.286700 24935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:37.289839 24937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:37.290446 24930 heartbeater.cc:347] Connected to a master server at 127.23.244.61:37851
I20240501 08:10:37.290620 24930 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.291343 24930 heartbeater.cc:511] Master 127.23.244.61:37851 requested a full tablet report, sending...
I20240501 08:10:37.292562 24623 ts_manager.cc:194] Registered new tserver with Master: cae247bed8d5431d9691590244e260a6 (127.23.244.2:38703)
I20240501 08:10:37.293660 24623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53268
I20240501 08:10:37.294446 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:37.294788 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:37.294915 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:37.294982 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551037294960 us; error 0 us; skew 500 ppm
I20240501 08:10:37.295197 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:37.296748 24528 webserver.cc:460] Webserver started at http://127.23.244.3:43147/ using document root <none> and password file <none>
I20240501 08:10:37.297016 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:37.297109 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:37.297261 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:37.298112 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.301450 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:37.303843 24942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.304256 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:10:37.304368 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84"
format_stamp: "Formatted at 2024-05-01 08:10:37 on dist-test-slave-fck4"
I20240501 08:10:37.304491 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:37.328258 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:37.329131 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:37.329810 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:37.330783 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:37.330878 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.330982 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:37.331068 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:37.353588 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:35905
I20240501 08:10:37.353613 25004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:35905 every 8 connection(s)
I20240501 08:10:37.372378 25006 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43933
I20240501 08:10:37.372572 25006 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.375301 25006 heartbeater.cc:511] Master 127.23.244.62:43933 requested a full tablet report, sending...
I20240501 08:10:37.376622 24558 ts_manager.cc:194] Registered new tserver with Master: 87c0d2bf0f3e469c9918c11fe4ba0f84 (127.23.244.3:35905)
I20240501 08:10:37.377787 25005 heartbeater.cc:347] Connected to a master server at 127.23.244.60:43347
I20240501 08:10:37.377894 25005 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.378764 25007 heartbeater.cc:347] Connected to a master server at 127.23.244.61:37851
I20240501 08:10:37.378914 25005 heartbeater.cc:511] Master 127.23.244.60:43347 requested a full tablet report, sending...
I20240501 08:10:37.379120 25007 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:37.379853 24688 ts_manager.cc:194] Registered new tserver with Master: 87c0d2bf0f3e469c9918c11fe4ba0f84 (127.23.244.3:35905)
I20240501 08:10:37.380411 25007 heartbeater.cc:511] Master 127.23.244.61:37851 requested a full tablet report, sending...
I20240501 08:10:37.381253 24623 ts_manager.cc:194] Registered new tserver with Master: 87c0d2bf0f3e469c9918c11fe4ba0f84 (127.23.244.3:35905)
I20240501 08:10:37.381700 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008738306s
I20240501 08:10:37.382212 24623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53278
I20240501 08:10:37.994629 24746 catalog_manager.cc:1259] Loaded cluster ID: 080d079aa96f4ad692bb9f077c411802
I20240501 08:10:37.994732 24746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: loading cluster ID for follower catalog manager: success
I20240501 08:10:37.996654 24746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: acquiring CA information for follower catalog manager: success
I20240501 08:10:37.997763 24766 catalog_manager.cc:1259] Loaded cluster ID: 080d079aa96f4ad692bb9f077c411802
I20240501 08:10:37.997854 24766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: loading cluster ID for follower catalog manager: success
I20240501 08:10:37.997931 24746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:10:37.999754 24766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: acquiring CA information for follower catalog manager: success
I20240501 08:10:38.000826 24766 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:10:38.188302 24854 heartbeater.cc:503] Master 127.23.244.61:37851 was elected leader, sending a full tablet report...
I20240501 08:10:38.295120 24930 heartbeater.cc:503] Master 127.23.244.61:37851 was elected leader, sending a full tablet report...
I20240501 08:10:38.383525 25007 heartbeater.cc:503] Master 127.23.244.61:37851 was elected leader, sending a full tablet report...
I20240501 08:10:38.989773 24757 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:39.440810 24528 test_util.cc:268] Using random seed: 94877577
I20240501 08:10:39.474769 24623 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53280:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:10:39.533458 24970 tablet_service.cc:1467] Processing CreateTablet for tablet fda3247aee56431b83930ede024d3aaa (DEFAULT_TABLE table=test-workload [id=8f59bd45c5db4608b72ed03ea5d770b5]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 08:10:39.534580 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda3247aee56431b83930ede024d3aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.534547 24893 tablet_service.cc:1467] Processing CreateTablet for tablet 52bb4f69c5aa4a3f8279cb73de273637 (DEFAULT_TABLE table=test-workload [id=8f59bd45c5db4608b72ed03ea5d770b5]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 08:10:39.534931 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52bb4f69c5aa4a3f8279cb73de273637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.537722 24815 tablet_service.cc:1467] Processing CreateTablet for tablet ac6bb109e6ca4ae99172474093f6c7b5 (DEFAULT_TABLE table=test-workload [id=8f59bd45c5db4608b72ed03ea5d770b5]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 08:10:39.538924 24815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6bb109e6ca4ae99172474093f6c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.539450 24814 tablet_service.cc:1467] Processing CreateTablet for tablet c9a6c25d18d54e0fb7db41d5fb7e06b2 (DEFAULT_TABLE table=test-workload [id=8f59bd45c5db4608b72ed03ea5d770b5]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 08:10:39.540037 24814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a6c25d18d54e0fb7db41d5fb7e06b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.548401 25040 tablet_bootstrap.cc:492] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: Bootstrap starting.
I20240501 08:10:39.550297 25041 tablet_bootstrap.cc:492] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: Bootstrap starting.
I20240501 08:10:39.552187 25040 tablet_bootstrap.cc:654] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:39.553164 25042 tablet_bootstrap.cc:492] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: Bootstrap starting.
I20240501 08:10:39.553795 25041 tablet_bootstrap.cc:654] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:39.555269 25042 tablet_bootstrap.cc:654] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:39.558804 25041 tablet_bootstrap.cc:492] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: No bootstrap required, opened a new log
I20240501 08:10:39.559119 25041 ts_tablet_manager.cc:1367] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 08:10:39.559262 25040 tablet_bootstrap.cc:492] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: No bootstrap required, opened a new log
I20240501 08:10:39.559420 25040 ts_tablet_manager.cc:1367] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 08:10:39.559708 25042 tablet_bootstrap.cc:492] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: No bootstrap required, opened a new log
I20240501 08:10:39.559823 25042 ts_tablet_manager.cc:1367] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 08:10:39.560230 25040 raft_consensus.cc:348] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae247bed8d5431d9691590244e260a6" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 38703 } }
I20240501 08:10:39.560166 25041 raft_consensus.cc:348] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35905 } }
I20240501 08:10:39.560407 25040 raft_consensus.cc:374] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:39.560364 25042 raft_consensus.cc:348] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.560480 25041 raft_consensus.cc:374] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:39.560560 25040 raft_consensus.cc:729] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae247bed8d5431d9691590244e260a6, State: Initialized, Role: FOLLOWER
I20240501 08:10:39.560592 25042 raft_consensus.cc:374] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:39.560673 25041 raft_consensus.cc:729] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c0d2bf0f3e469c9918c11fe4ba0f84, State: Initialized, Role: FOLLOWER
I20240501 08:10:39.560775 25042 raft_consensus.cc:729] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d939bbd4dc452e9b39d7bfe422f17e, State: Initialized, Role: FOLLOWER
I20240501 08:10:39.560923 25040 consensus_queue.cc:260] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae247bed8d5431d9691590244e260a6" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 38703 } }
I20240501 08:10:39.561057 25042 consensus_queue.cc:260] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.561306 25040 raft_consensus.cc:388] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:39.561327 25042 raft_consensus.cc:388] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:39.561460 25040 raft_consensus.cc:482] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:39.561463 25042 raft_consensus.cc:482] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:39.561654 25040 raft_consensus.cc:3037] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:39.561697 25042 raft_consensus.cc:3037] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:39.560925 25041 consensus_queue.cc:260] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35905 } }
I20240501 08:10:39.563618 25041 raft_consensus.cc:388] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:39.563748 25041 raft_consensus.cc:482] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:39.563882 25041 raft_consensus.cc:3037] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:39.564519 25042 raft_consensus.cc:504] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.564786 25042 leader_election.cc:302] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d939bbd4dc452e9b39d7bfe422f17e; no voters:
I20240501 08:10:39.564800 25040 raft_consensus.cc:504] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae247bed8d5431d9691590244e260a6" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 38703 } }
I20240501 08:10:39.565078 25040 leader_election.cc:302] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae247bed8d5431d9691590244e260a6; no voters:
I20240501 08:10:39.567317 25041 raft_consensus.cc:504] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35905 } }
I20240501 08:10:39.567632 25041 leader_election.cc:302] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c0d2bf0f3e469c9918c11fe4ba0f84; no voters:
I20240501 08:10:39.568953 25047 raft_consensus.cc:2781] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:39.569496 25040 leader_election.cc:288] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:39.571683 25040 ts_tablet_manager.cc:1398] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: Time spent starting tablet: real 0.011s user 0.002s sys 0.008s
I20240501 08:10:39.572489 25047 raft_consensus.cc:686] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 1 LEADER]: Becoming Leader. State: Replica: cae247bed8d5431d9691590244e260a6, State: Running, Role: LEADER
I20240501 08:10:39.573896 25047 consensus_queue.cc:237] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae247bed8d5431d9691590244e260a6" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 38703 } }
I20240501 08:10:39.575101 25042 leader_election.cc:288] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:39.575354 25046 raft_consensus.cc:2781] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:39.577051 25046 raft_consensus.cc:686] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 LEADER]: Becoming Leader. State: Replica: 42d939bbd4dc452e9b39d7bfe422f17e, State: Running, Role: LEADER
I20240501 08:10:39.577376 25041 leader_election.cc:288] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:39.578294 25048 raft_consensus.cc:2781] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:39.579733 25042 ts_tablet_manager.cc:1398] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: Time spent starting tablet: real 0.020s user 0.007s sys 0.013s
I20240501 08:10:39.579999 25042 tablet_bootstrap.cc:492] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: Bootstrap starting.
I20240501 08:10:39.581353 25042 tablet_bootstrap.cc:654] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:39.584513 24623 catalog_manager.cc:5484] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cae247bed8d5431d9691590244e260a6 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "cae247bed8d5431d9691590244e260a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae247bed8d5431d9691590244e260a6" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:39.587060 25041 ts_tablet_manager.cc:1398] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: Time spent starting tablet: real 0.028s user 0.008s sys 0.014s
I20240501 08:10:39.577543 25046 consensus_queue.cc:237] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.591166 24622 catalog_manager.cc:5484] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e reported cstate change: term changed from 0 to 1, leader changed from <none> to 42d939bbd4dc452e9b39d7bfe422f17e (127.23.244.1). New cstate: current_term: 1 leader_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:39.594727 25048 raft_consensus.cc:686] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 1 LEADER]: Becoming Leader. State: Replica: 87c0d2bf0f3e469c9918c11fe4ba0f84, State: Running, Role: LEADER
I20240501 08:10:39.595117 25048 consensus_queue.cc:237] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35905 } }
I20240501 08:10:39.600337 24622 catalog_manager.cc:5484] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87c0d2bf0f3e469c9918c11fe4ba0f84 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c0d2bf0f3e469c9918c11fe4ba0f84" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35905 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:39.605911 25042 tablet_bootstrap.cc:492] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: No bootstrap required, opened a new log
I20240501 08:10:39.606218 25042 ts_tablet_manager.cc:1367] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20240501 08:10:39.607910 25042 raft_consensus.cc:348] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.608193 25042 raft_consensus.cc:374] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:39.608325 25042 raft_consensus.cc:729] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d939bbd4dc452e9b39d7bfe422f17e, State: Initialized, Role: FOLLOWER
I20240501 08:10:39.608636 25042 consensus_queue.cc:260] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.608911 25042 raft_consensus.cc:388] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:39.609098 25042 raft_consensus.cc:482] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:39.609227 25042 raft_consensus.cc:3037] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:39.612629 25042 raft_consensus.cc:504] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.612962 25042 leader_election.cc:302] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d939bbd4dc452e9b39d7bfe422f17e; no voters:
I20240501 08:10:39.613242 25042 leader_election.cc:288] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:39.613292 25052 raft_consensus.cc:2781] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:39.614085 25042 ts_tablet_manager.cc:1398] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:10:39.613502 25052 raft_consensus.cc:686] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 LEADER]: Becoming Leader. State: Replica: 42d939bbd4dc452e9b39d7bfe422f17e, State: Running, Role: LEADER
I20240501 08:10:39.614830 25052 consensus_queue.cc:237] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } }
I20240501 08:10:39.619422 24622 catalog_manager.cc:5484] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e reported cstate change: term changed from 0 to 1, leader changed from <none> to 42d939bbd4dc452e9b39d7bfe422f17e (127.23.244.1). New cstate: current_term: 1 leader_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d939bbd4dc452e9b39d7bfe422f17e" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 38031 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:39.645777 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:10:39.655588 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:39.655979 24528 tablet_replica.cc:330] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e: stopping tablet replica
I20240501 08:10:39.656423 24528 raft_consensus.cc:2229] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:39.656674 24528 raft_consensus.cc:2258] T c9a6c25d18d54e0fb7db41d5fb7e06b2 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.658143 24528 tablet_replica.cc:330] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e: stopping tablet replica
I20240501 08:10:39.658308 24528 raft_consensus.cc:2229] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:39.658452 24528 raft_consensus.cc:2258] T ac6bb109e6ca4ae99172474093f6c7b5 P 42d939bbd4dc452e9b39d7bfe422f17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.663600 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:10:39.668677 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:10:39.676977 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:39.677376 24528 tablet_replica.cc:330] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6: stopping tablet replica
I20240501 08:10:39.677561 24528 raft_consensus.cc:2229] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:39.677731 24528 raft_consensus.cc:2258] T 52bb4f69c5aa4a3f8279cb73de273637 P cae247bed8d5431d9691590244e260a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.691825 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:10:39.694370 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:10:39.702601 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:39.702965 24528 tablet_replica.cc:330] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84: stopping tablet replica
I20240501 08:10:39.703186 24528 raft_consensus.cc:2229] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:39.703356 24528 raft_consensus.cc:2258] T fda3247aee56431b83930ede024d3aaa P 87c0d2bf0f3e469c9918c11fe4ba0f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.707602 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:10:39.710268 24528 master.cc:537] Master@127.23.244.62:43933 shutting down...
I20240501 08:10:39.718575 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:39.718788 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.718902 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P bf6eaaa2f6084a22a04a4591e3bc7dc6: stopping tablet replica
I20240501 08:10:39.733513 24528 master.cc:552] Master@127.23.244.62:43933 shutdown complete.
I20240501 08:10:39.738030 24528 master.cc:537] Master@127.23.244.61:37851 shutting down...
I20240501 08:10:39.745048 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:39.745481 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.745608 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P b77b8451c1de4472a1fdbb549713f6a7: stopping tablet replica
I20240501 08:10:39.760426 24528 master.cc:552] Master@127.23.244.61:37851 shutdown complete.
I20240501 08:10:39.764519 24528 master.cc:537] Master@127.23.244.60:43347 shutting down...
I20240501 08:10:39.772024 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:39.772231 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:39.772397 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3edac21ac7f4e44b24d86df35574f59: stopping tablet replica
I20240501 08:10:39.786154 24528 master.cc:552] Master@127.23.244.60:43347 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3084 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 08:10:39.798548 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:33625
I20240501 08:10:39.799027 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.801368 25063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.802037 25064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.802778 25066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.803720 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:39.803966 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:39.804050 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:39.804109 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551039804091 us; error 0 us; skew 500 ppm
I20240501 08:10:39.804278 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.805512 24528 webserver.cc:460] Webserver started at http://127.23.244.62:38045/ using document root <none> and password file <none>
I20240501 08:10:39.805757 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.805836 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.806005 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.806681 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "43287369b0614e0f93647b54f666c9ce"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.809132 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:10:39.810604 25071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.810936 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:39.811040 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "43287369b0614e0f93647b54f666c9ce"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.811153 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.835938 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:39.836598 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.850734 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:33625
I20240501 08:10:39.850765 25122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:33625 every 8 connection(s)
I20240501 08:10:39.852026 25123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.855902 25123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce: Bootstrap starting.
I20240501 08:10:39.857201 25123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:39.858669 25123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce: No bootstrap required, opened a new log
I20240501 08:10:39.859329 25123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER }
I20240501 08:10:39.859475 25123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:39.859573 25123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43287369b0614e0f93647b54f666c9ce, State: Initialized, Role: FOLLOWER
I20240501 08:10:39.859783 25123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER }
I20240501 08:10:39.859926 25123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:39.860000 25123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:39.860113 25123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:39.861861 25123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER }
I20240501 08:10:39.862083 25123 leader_election.cc:302] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43287369b0614e0f93647b54f666c9ce; no voters:
I20240501 08:10:39.862486 25123 leader_election.cc:288] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:39.862635 25126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:39.863134 25126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 1 LEADER]: Becoming Leader. State: Replica: 43287369b0614e0f93647b54f666c9ce, State: Running, Role: LEADER
I20240501 08:10:39.863440 25123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:39.863389 25126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER }
I20240501 08:10:39.864313 25128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43287369b0614e0f93647b54f666c9ce. Latest consensus state: current_term: 1 leader_uuid: "43287369b0614e0f93647b54f666c9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER } }
I20240501 08:10:39.864291 25127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43287369b0614e0f93647b54f666c9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43287369b0614e0f93647b54f666c9ce" member_type: VOTER } }
I20240501 08:10:39.864533 25128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:39.864648 25127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:39.866053 25131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:39.868106 25131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:39.869393 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:10:39.871315 25131 catalog_manager.cc:1347] Generated new cluster ID: 22cee95c74684bd7b3f5fdf4c1370558
I20240501 08:10:39.871400 25131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:39.881466 25131 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:39.882179 25131 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:39.890627 25131 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce: Generated new TSK 0
I20240501 08:10:39.890853 25131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:39.902223 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.905107 25144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.905784 25145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.906417 25147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.906740 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:39.907009 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:39.907068 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:39.907120 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551039907094 us; error 0 us; skew 500 ppm
I20240501 08:10:39.907232 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.908421 24528 webserver.cc:460] Webserver started at http://127.23.244.1:37853/ using document root <none> and password file <none>
I20240501 08:10:39.908630 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.908706 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.908854 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.909459 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.911863 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:10:39.913439 25152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.913810 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:39.913897 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.914011 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.922078 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:39.922581 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.923074 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:39.923837 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:39.923897 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.923972 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:39.924012 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.939064 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:42463
I20240501 08:10:39.939085 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:42463 every 8 connection(s)
I20240501 08:10:39.940701 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.943850 25219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.944654 25220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.947264 25222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.948532 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:39.948834 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:39.948936 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:39.949004 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551039948976 us; error 0 us; skew 500 ppm
I20240501 08:10:39.949187 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.950817 24528 webserver.cc:460] Webserver started at http://127.23.244.2:36571/ using document root <none> and password file <none>
I20240501 08:10:39.951114 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.951213 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.951383 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.952250 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.953927 25215 heartbeater.cc:347] Connected to a master server at 127.23.244.62:33625
I20240501 08:10:39.954107 25215 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:39.954550 25215 heartbeater.cc:511] Master 127.23.244.62:33625 requested a full tablet report, sending...
I20240501 08:10:39.955366 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:10:39.955646 25088 ts_manager.cc:194] Registered new tserver with Master: 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463)
I20240501 08:10:39.957142 25088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36760
I20240501 08:10:39.957484 25227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.957896 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 08:10:39.958050 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:39.958266 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.970829 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:39.971352 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.971863 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:39.972646 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:39.972746 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.972831 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:39.972883 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.986603 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:43569
I20240501 08:10:39.986663 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:43569 every 8 connection(s)
I20240501 08:10:39.988065 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.991503 25293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.992348 25294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.999338 25290 heartbeater.cc:347] Connected to a master server at 127.23.244.62:33625
W20240501 08:10:39.999367 25296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.999473 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:39.999563 25290 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.000136 25290 heartbeater.cc:511] Master 127.23.244.62:33625 requested a full tablet report, sending...
I20240501 08:10:40.000339 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:40.000420 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:40.000505 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551040000465 us; error 0 us; skew 500 ppm
I20240501 08:10:40.000672 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.001149 25088 ts_manager.cc:194] Registered new tserver with Master: 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569)
I20240501 08:10:40.002030 24528 webserver.cc:460] Webserver started at http://127.23.244.3:41649/ using document root <none> and password file <none>
I20240501 08:10:40.002183 25088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36776
I20240501 08:10:40.002324 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.002444 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.002596 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.003393 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "8afbcfee573c49a7a3185af453c8b0fe"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:40.005954 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 08:10:40.007479 25301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.007818 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:40.007912 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "8afbcfee573c49a7a3185af453c8b0fe"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-fck4"
I20240501 08:10:40.008060 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.041868 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:40.042495 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.043032 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:40.043812 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:40.043888 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.043960 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:40.044005 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.058099 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:33535
I20240501 08:10:40.058135 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:33535 every 8 connection(s)
I20240501 08:10:40.065250 25364 heartbeater.cc:347] Connected to a master server at 127.23.244.62:33625
I20240501 08:10:40.065388 25364 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.065759 25364 heartbeater.cc:511] Master 127.23.244.62:33625 requested a full tablet report, sending...
I20240501 08:10:40.066689 25088 ts_manager.cc:194] Registered new tserver with Master: 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:40.067104 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007951305s
I20240501 08:10:40.067662 25088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36780
I20240501 08:10:40.958781 25215 heartbeater.cc:503] Master 127.23.244.62:33625 was elected leader, sending a full tablet report...
I20240501 08:10:41.003861 25290 heartbeater.cc:503] Master 127.23.244.62:33625 was elected leader, sending a full tablet report...
I20240501 08:10:41.069264 25364 heartbeater.cc:503] Master 127.23.244.62:33625 was elected leader, sending a full tablet report...
I20240501 08:10:41.095474 24528 test_util.cc:268] Using random seed: 96532236
I20240501 08:10:41.111299 25088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36792:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:10:41.112015 25088 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:10:41.137941 25255 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea29fe6dd4c424c80c6026f878676ea (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:10:41.138597 25255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea29fe6dd4c424c80c6026f878676ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.138638 25252 tablet_service.cc:1467] Processing CreateTablet for tablet bede93e312574be3927b1e35d259ca57 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:10:41.139047 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bede93e312574be3927b1e35d259ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.139506 25248 tablet_service.cc:1467] Processing CreateTablet for tablet 343e4dfc3c484385a9eb31ee16c5431f (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:10:41.139991 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343e4dfc3c484385a9eb31ee16c5431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.141146 25253 tablet_service.cc:1467] Processing CreateTablet for tablet f5795a466f3c4c938eb00d01fe3c0ed2 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:10:41.142141 25253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5795a466f3c4c938eb00d01fe3c0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.137984 25254 tablet_service.cc:1467] Processing CreateTablet for tablet ec175c6ee540428a987372f7e5f629f1 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:10:41.144280 25329 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea29fe6dd4c424c80c6026f878676ea (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:10:41.145028 25328 tablet_service.cc:1467] Processing CreateTablet for tablet ec175c6ee540428a987372f7e5f629f1 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:10:41.145895 25329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea29fe6dd4c424c80c6026f878676ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.147061 25254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec175c6ee540428a987372f7e5f629f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.146932 25250 tablet_service.cc:1467] Processing CreateTablet for tablet b2e1ae299e884a299ad04395af4ffb34 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:10:41.147375 25327 tablet_service.cc:1467] Processing CreateTablet for tablet f5795a466f3c4c938eb00d01fe3c0ed2 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:10:41.147429 25249 tablet_service.cc:1467] Processing CreateTablet for tablet 0b83f73f70e24f3ca20628cda62e8925 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:10:41.148895 25328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec175c6ee540428a987372f7e5f629f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.149319 25327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5795a466f3c4c938eb00d01fe3c0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.149750 25250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e1ae299e884a299ad04395af4ffb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.149981 25249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b83f73f70e24f3ca20628cda62e8925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.151219 25326 tablet_service.cc:1467] Processing CreateTablet for tablet bede93e312574be3927b1e35d259ca57 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:10:41.151247 25325 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3437ee7ff472caffbc61fa5f5fb38 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:10:41.153482 25324 tablet_service.cc:1467] Processing CreateTablet for tablet b2e1ae299e884a299ad04395af4ffb34 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:10:41.154284 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 343e4dfc3c484385a9eb31ee16c5431f (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:10:41.154796 25323 tablet_service.cc:1467] Processing CreateTablet for tablet 0b83f73f70e24f3ca20628cda62e8925 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:10:41.155261 25251 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3437ee7ff472caffbc61fa5f5fb38 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:10:41.156119 25251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3437ee7ff472caffbc61fa5f5fb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.158545 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bede93e312574be3927b1e35d259ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.158783 25325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3437ee7ff472caffbc61fa5f5fb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.160871 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343e4dfc3c484385a9eb31ee16c5431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.161175 25323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b83f73f70e24f3ca20628cda62e8925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.161460 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e1ae299e884a299ad04395af4ffb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.188372 25180 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea29fe6dd4c424c80c6026f878676ea (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 08:10:41.189913 25179 tablet_service.cc:1467] Processing CreateTablet for tablet ec175c6ee540428a987372f7e5f629f1 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 08:10:41.192157 25180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea29fe6dd4c424c80c6026f878676ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.194397 25176 tablet_service.cc:1467] Processing CreateTablet for tablet b6d3437ee7ff472caffbc61fa5f5fb38 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 08:10:41.195363 25179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec175c6ee540428a987372f7e5f629f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.196683 25174 tablet_service.cc:1467] Processing CreateTablet for tablet 0b83f73f70e24f3ca20628cda62e8925 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 08:10:41.197419 25175 tablet_service.cc:1467] Processing CreateTablet for tablet b2e1ae299e884a299ad04395af4ffb34 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 08:10:41.198861 25176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d3437ee7ff472caffbc61fa5f5fb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.199720 25178 tablet_service.cc:1467] Processing CreateTablet for tablet f5795a466f3c4c938eb00d01fe3c0ed2 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 08:10:41.201408 25178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5795a466f3c4c938eb00d01fe3c0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.203119 25177 tablet_service.cc:1467] Processing CreateTablet for tablet bede93e312574be3927b1e35d259ca57 (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 08:10:41.203883 25174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b83f73f70e24f3ca20628cda62e8925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.203933 25173 tablet_service.cc:1467] Processing CreateTablet for tablet 343e4dfc3c484385a9eb31ee16c5431f (DEFAULT_TABLE table=test-workload [id=7238d58aa21847a7a974b2f57c1c3c8e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 08:10:41.204895 25175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e1ae299e884a299ad04395af4ffb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.205744 25384 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.206379 25177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bede93e312574be3927b1e35d259ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.209607 25173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343e4dfc3c484385a9eb31ee16c5431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.212397 25385 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.229414 25386 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.230836 25384 tablet_bootstrap.cc:654] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.232564 25385 tablet_bootstrap.cc:654] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.232703 25386 tablet_bootstrap.cc:654] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.235702 25386 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.235857 25386 ts_tablet_manager.cc:1367] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 08:10:41.236485 25386 raft_consensus.cc:348] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.236665 25386 raft_consensus.cc:374] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.236836 25386 raft_consensus.cc:729] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.237061 25386 consensus_queue.cc:260] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.237910 25386 ts_tablet_manager.cc:1398] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:41.238190 25386 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.238740 25385 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.239042 25385 ts_tablet_manager.cc:1367] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.000s
I20240501 08:10:41.239975 25386 tablet_bootstrap.cc:654] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.240178 25385 raft_consensus.cc:348] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.240680 25385 raft_consensus.cc:374] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.240916 25385 raft_consensus.cc:729] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.241353 25385 consensus_queue.cc:260] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.243299 25384 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.243530 25384 ts_tablet_manager.cc:1367] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.012s
I20240501 08:10:41.244467 25384 raft_consensus.cc:348] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.244664 25384 raft_consensus.cc:374] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.244753 25385 ts_tablet_manager.cc:1398] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:41.244793 25384 raft_consensus.cc:729] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.245155 25385 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.245203 25384 consensus_queue.cc:260] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.245857 25386 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.246656 25386 ts_tablet_manager.cc:1367] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:10:41.247114 25384 ts_tablet_manager.cc:1398] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20240501 08:10:41.247457 25384 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.247907 25385 tablet_bootstrap.cc:654] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.248826 25384 tablet_bootstrap.cc:654] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.249743 25386 raft_consensus.cc:348] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.250154 25386 raft_consensus.cc:374] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.250322 25386 raft_consensus.cc:729] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.250766 25386 consensus_queue.cc:260] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.251564 25386 ts_tablet_manager.cc:1398] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:10:41.254743 25384 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.254966 25384 ts_tablet_manager.cc:1367] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240501 08:10:41.255395 25386 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.255822 25384 raft_consensus.cc:348] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.256126 25384 raft_consensus.cc:374] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.256278 25384 raft_consensus.cc:729] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.256609 25384 consensus_queue.cc:260] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.257135 25386 tablet_bootstrap.cc:654] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.257373 25384 ts_tablet_manager.cc:1398] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:41.257701 25384 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.259555 25384 tablet_bootstrap.cc:654] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.259974 25385 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.260223 25385 ts_tablet_manager.cc:1367] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.003s
I20240501 08:10:41.261138 25385 raft_consensus.cc:348] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.261463 25385 raft_consensus.cc:374] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.261600 25385 raft_consensus.cc:729] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.261948 25385 consensus_queue.cc:260] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.262552 25385 ts_tablet_manager.cc:1398] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:41.262888 25385 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.263737 25386 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.264040 25386 ts_tablet_manager.cc:1367] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 08:10:41.264633 25385 tablet_bootstrap.cc:654] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.265089 25386 raft_consensus.cc:348] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.265457 25386 raft_consensus.cc:374] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.265646 25386 raft_consensus.cc:729] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.266074 25386 consensus_queue.cc:260] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.266981 25386 ts_tablet_manager.cc:1398] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:41.267410 25386 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.270454 25386 tablet_bootstrap.cc:654] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.272879 25384 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.272903 25385 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.273036 25386 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.273072 25384 ts_tablet_manager.cc:1367] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.011s
I20240501 08:10:41.273075 25385 ts_tablet_manager.cc:1367] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 08:10:41.273208 25386 ts_tablet_manager.cc:1367] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 08:10:41.273944 25384 raft_consensus.cc:348] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.274189 25384 raft_consensus.cc:374] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.273957 25386 raft_consensus.cc:348] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.273965 25385 raft_consensus.cc:348] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.274288 25384 raft_consensus.cc:729] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.274379 25385 raft_consensus.cc:374] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.274399 25386 raft_consensus.cc:374] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.274478 25385 raft_consensus.cc:729] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.274497 25386 raft_consensus.cc:729] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.274628 25384 consensus_queue.cc:260] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.274755 25385 consensus_queue.cc:260] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.274756 25386 consensus_queue.cc:260] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.275332 25385 ts_tablet_manager.cc:1398] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.275593 25386 ts_tablet_manager.cc:1398] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.275653 25385 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.275918 25384 ts_tablet_manager.cc:1398] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:41.275947 25386 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.276197 25384 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.277509 25385 tablet_bootstrap.cc:654] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.277894 25384 tablet_bootstrap.cc:654] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.277951 25386 tablet_bootstrap.cc:654] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.283677 25386 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.283917 25386 ts_tablet_manager.cc:1367] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 08:10:41.284862 25386 raft_consensus.cc:348] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.285166 25386 raft_consensus.cc:374] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.285333 25386 raft_consensus.cc:729] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.285652 25386 consensus_queue.cc:260] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.286420 25386 ts_tablet_manager.cc:1398] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.286732 25386 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.288566 25386 tablet_bootstrap.cc:654] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.290833 25385 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.291013 25385 ts_tablet_manager.cc:1367] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.003s
I20240501 08:10:41.291234 25384 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.291390 25384 ts_tablet_manager.cc:1367] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240501 08:10:41.291816 25385 raft_consensus.cc:348] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.292070 25384 raft_consensus.cc:348] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.292654 25385 raft_consensus.cc:374] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.292654 25384 raft_consensus.cc:374] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.292788 25385 raft_consensus.cc:729] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.292809 25384 raft_consensus.cc:729] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.293084 25385 consensus_queue.cc:260] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.293110 25384 consensus_queue.cc:260] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.293330 25386 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.293529 25386 ts_tablet_manager.cc:1367] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 08:10:41.293684 25384 ts_tablet_manager.cc:1398] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.294049 25384 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.294225 25386 raft_consensus.cc:348] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.294476 25386 raft_consensus.cc:374] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.294586 25386 raft_consensus.cc:729] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.294888 25386 consensus_queue.cc:260] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.295473 25386 ts_tablet_manager.cc:1398] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:10:41.295778 25384 tablet_bootstrap.cc:654] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.295970 25385 ts_tablet_manager.cc:1398] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:10:41.296502 25386 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.297888 25385 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.298192 25386 tablet_bootstrap.cc:654] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.300001 25385 tablet_bootstrap.cc:654] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.302024 25386 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.302165 25386 ts_tablet_manager.cc:1367] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:10:41.302670 25385 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.302846 25385 ts_tablet_manager.cc:1367] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:10:41.302800 25386 raft_consensus.cc:348] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.303010 25386 raft_consensus.cc:374] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.303103 25386 raft_consensus.cc:729] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.303290 25386 consensus_queue.cc:260] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.303640 25385 raft_consensus.cc:348] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.304252 25385 raft_consensus.cc:374] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.304286 25386 ts_tablet_manager.cc:1398] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 08:10:41.304342 25384 tablet_bootstrap.cc:492] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.304364 25385 raft_consensus.cc:729] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.304479 25384 ts_tablet_manager.cc:1367] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240501 08:10:41.304590 25386 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: Bootstrap starting.
I20240501 08:10:41.304610 25385 consensus_queue.cc:260] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.305274 25384 raft_consensus.cc:348] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.305460 25384 raft_consensus.cc:374] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.305563 25384 raft_consensus.cc:729] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.305806 25384 consensus_queue.cc:260] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.306355 25386 tablet_bootstrap.cc:654] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.307047 25384 ts_tablet_manager.cc:1398] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.307262 25385 ts_tablet_manager.cc:1398] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:10:41.307312 25384 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.309232 25384 tablet_bootstrap.cc:654] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.310091 25385 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.312243 25386 tablet_bootstrap.cc:492] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: No bootstrap required, opened a new log
I20240501 08:10:41.312500 25385 tablet_bootstrap.cc:654] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.312561 25386 ts_tablet_manager.cc:1367] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 08:10:41.313923 25386 raft_consensus.cc:348] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.314154 25386 raft_consensus.cc:374] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.314296 25386 raft_consensus.cc:729] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.314538 25386 consensus_queue.cc:260] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.315585 25386 ts_tablet_manager.cc:1398] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:41.317931 25384 tablet_bootstrap.cc:492] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.318181 25384 ts_tablet_manager.cc:1367] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 08:10:41.319238 25384 raft_consensus.cc:348] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.319577 25384 raft_consensus.cc:374] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.319686 25384 raft_consensus.cc:729] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.320055 25384 consensus_queue.cc:260] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.320802 25384 ts_tablet_manager.cc:1398] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:10:41.321166 25384 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.321640 25385 tablet_bootstrap.cc:492] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.321822 25385 ts_tablet_manager.cc:1367] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20240501 08:10:41.322633 25385 raft_consensus.cc:348] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.322885 25385 raft_consensus.cc:374] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.322986 25385 raft_consensus.cc:729] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.323040 25384 tablet_bootstrap.cc:654] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.323369 25385 consensus_queue.cc:260] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.324781 25385 ts_tablet_manager.cc:1398] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:10:41.325127 25385 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.325479 25384 tablet_bootstrap.cc:492] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.325789 25384 ts_tablet_manager.cc:1367] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:10:41.326674 25384 raft_consensus.cc:348] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.326930 25385 tablet_bootstrap.cc:654] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.326927 25384 raft_consensus.cc:374] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.327143 25384 raft_consensus.cc:729] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.327512 25384 consensus_queue.cc:260] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.328646 25384 ts_tablet_manager.cc:1398] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:10:41.328992 25384 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Bootstrap starting.
I20240501 08:10:41.329905 25385 tablet_bootstrap.cc:492] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.330118 25385 ts_tablet_manager.cc:1367] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:41.330780 25384 tablet_bootstrap.cc:654] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.331038 25385 raft_consensus.cc:348] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.331292 25385 raft_consensus.cc:374] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.331403 25385 raft_consensus.cc:729] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.331660 25385 consensus_queue.cc:260] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.332512 25385 ts_tablet_manager.cc:1398] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:10:41.332938 25385 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: Bootstrap starting.
I20240501 08:10:41.334031 25393 raft_consensus.cc:482] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.334177 25393 raft_consensus.cc:504] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.335121 25393 leader_election.cc:288] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.335361 25385 tablet_bootstrap.cc:654] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.339269 25384 tablet_bootstrap.cc:492] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: No bootstrap required, opened a new log
I20240501 08:10:41.339514 25384 ts_tablet_manager.cc:1367] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240501 08:10:41.341810 25384 raft_consensus.cc:348] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.342209 25384 raft_consensus.cc:374] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.342329 25384 raft_consensus.cc:729] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.342657 25384 consensus_queue.cc:260] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.344383 25384 ts_tablet_manager.cc:1398] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:10:41.344853 25385 tablet_bootstrap.cc:492] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: No bootstrap required, opened a new log
I20240501 08:10:41.345009 25385 ts_tablet_manager.cc:1367] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 08:10:41.345654 25385 raft_consensus.cc:348] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.345903 25385 raft_consensus.cc:374] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.346011 25385 raft_consensus.cc:729] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.346304 25385 consensus_queue.cc:260] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.347375 25385 ts_tablet_manager.cc:1398] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:41.348973 25393 raft_consensus.cc:482] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.349210 25393 raft_consensus.cc:504] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.349880 25393 leader_election.cc:288] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.356462 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe" is_pre_election: true
I20240501 08:10:41.356557 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b83f73f70e24f3ca20628cda62e8925" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe" is_pre_election: true
I20240501 08:10:41.356935 25338 raft_consensus.cc:2454] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.356954 25339 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.357492 25231 leader_election.cc:302] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.357431 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.357431 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b83f73f70e24f3ca20628cda62e8925" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.357792 25190 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.357825 25189 raft_consensus.cc:2454] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.357908 25231 leader_election.cc:302] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.358160 25393 raft_consensus.cc:2781] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.358269 25393 raft_consensus.cc:482] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.358409 25393 raft_consensus.cc:3037] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.360566 25393 raft_consensus.cc:504] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.361315 25390 raft_consensus.cc:482] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.361557 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b83f73f70e24f3ca20628cda62e8925" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.361454 25390 raft_consensus.cc:504] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.361737 25189 raft_consensus.cc:3037] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.362540 25390 leader_election.cc:288] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.361192 25393 leader_election.cc:288] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.362808 25393 raft_consensus.cc:2781] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.362923 25393 raft_consensus.cc:482] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.363030 25393 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.365080 25189 raft_consensus.cc:2454] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.365921 25393 raft_consensus.cc:504] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.366600 25393 leader_election.cc:288] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.367566 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b83f73f70e24f3ca20628cda62e8925" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:41.368091 25339 raft_consensus.cc:3037] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.368257 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:41.368472 25338 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.368846 25416 raft_consensus.cc:482] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.369024 25416 raft_consensus.cc:504] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.370023 25416 leader_election.cc:288] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.371587 25339 raft_consensus.cc:2454] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.371670 25338 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.372115 25231 leader_election.cc:302] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.372473 25231 leader_election.cc:302] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.372532 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec175c6ee540428a987372f7e5f629f1" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe" is_pre_election: true
I20240501 08:10:41.372803 25393 raft_consensus.cc:2781] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.372866 25338 raft_consensus.cc:2454] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.374269 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec175c6ee540428a987372f7e5f629f1" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.374529 25190 raft_consensus.cc:2454] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.375031 25230 leader_election.cc:302] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.375233 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.375568 25189 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.378198 25416 raft_consensus.cc:2781] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.379712 25189 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.381978 25416 raft_consensus.cc:686] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Running, Role: LEADER
I20240501 08:10:41.382117 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5795a466f3c4c938eb00d01fe3c0ed2" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" is_pre_election: true
I20240501 08:10:41.382495 25265 raft_consensus.cc:2454] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 0.
I20240501 08:10:41.382382 25416 consensus_queue.cc:237] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.383054 25156 leader_election.cc:302] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.383646 25420 raft_consensus.cc:2781] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.383769 25420 raft_consensus.cc:482] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.383914 25420 raft_consensus.cc:3037] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.384836 25390 raft_consensus.cc:2781] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.385051 25390 raft_consensus.cc:482] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.385183 25390 raft_consensus.cc:3037] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.388307 25393 raft_consensus.cc:686] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Running, Role: LEADER
I20240501 08:10:41.388710 25393 consensus_queue.cc:237] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.389101 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5795a466f3c4c938eb00d01fe3c0ed2" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe" is_pre_election: true
I20240501 08:10:41.389341 25338 raft_consensus.cc:2454] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 0.
I20240501 08:10:41.391566 25390 raft_consensus.cc:504] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.391616 25420 raft_consensus.cc:504] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.392328 25390 leader_election.cc:288] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 1 election: Requested vote from peers 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.392613 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5795a466f3c4c938eb00d01fe3c0ed2" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
I20240501 08:10:41.393286 25265 raft_consensus.cc:3037] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.393350 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec175c6ee540428a987372f7e5f629f1" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.393496 25189 raft_consensus.cc:3037] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.393319 25088 catalog_manager.cc:5484] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:41.395704 25189 raft_consensus.cc:2454] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.396108 25420 leader_election.cc:288] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.396100 25230 leader_election.cc:302] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.396332 25420 raft_consensus.cc:2781] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.396447 25420 raft_consensus.cc:686] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Running, Role: LEADER
I20240501 08:10:41.396672 25420 consensus_queue.cc:237] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.397091 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec175c6ee540428a987372f7e5f629f1" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:41.397672 25265 raft_consensus.cc:2454] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 1.
I20240501 08:10:41.397707 25338 raft_consensus.cc:3037] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.398836 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5795a466f3c4c938eb00d01fe3c0ed2" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:41.399067 25339 raft_consensus.cc:3037] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.399665 25088 catalog_manager.cc:5484] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:41.400230 25338 raft_consensus.cc:2454] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.400317 25088 catalog_manager.cc:5484] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:41.400897 25156 leader_election.cc:302] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.401167 25390 raft_consensus.cc:2781] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.401650 25390 raft_consensus.cc:686] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 LEADER]: Becoming Leader. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Running, Role: LEADER
I20240501 08:10:41.401721 25339 raft_consensus.cc:2454] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 1.
I20240501 08:10:41.402001 25390 consensus_queue.cc:237] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.404559 25088 catalog_manager.cc:5484] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d reported cstate change: term changed from 0 to 1, leader changed from <none> to 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1). New cstate: current_term: 1 leader_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:41.433195 25392 raft_consensus.cc:482] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.433387 25392 raft_consensus.cc:504] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.434770 25392 leader_election.cc:288] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569)
I20240501 08:10:41.441236 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bede93e312574be3927b1e35d259ca57" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.441319 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bede93e312574be3927b1e35d259ca57" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" is_pre_election: true
I20240501 08:10:41.442042 25189 raft_consensus.cc:2454] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.442044 25265 raft_consensus.cc:2454] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.442476 25305 leader_election.cc:302] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.442875 25392 raft_consensus.cc:2781] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.443022 25392 raft_consensus.cc:482] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.443127 25392 raft_consensus.cc:3037] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.445089 25392 raft_consensus.cc:504] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.445658 25392 leader_election.cc:288] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569)
I20240501 08:10:41.445958 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bede93e312574be3927b1e35d259ca57" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.446058 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bede93e312574be3927b1e35d259ca57" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
I20240501 08:10:41.446183 25189 raft_consensus.cc:3037] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.446233 25265 raft_consensus.cc:3037] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.449154 25265 raft_consensus.cc:2454] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.449155 25189 raft_consensus.cc:2454] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.449517 25305 leader_election.cc:302] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.449782 25392 raft_consensus.cc:2781] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.450165 25392 raft_consensus.cc:686] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Becoming Leader. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Running, Role: LEADER
I20240501 08:10:41.450467 25392 consensus_queue.cc:237] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.452960 25088 catalog_manager.cc:5484] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3). New cstate: current_term: 1 leader_uuid: "8afbcfee573c49a7a3185af453c8b0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:41.510164 25392 raft_consensus.cc:482] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.510422 25392 raft_consensus.cc:504] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.511147 25392 leader_election.cc:288] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569), 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463)
I20240501 08:10:41.511719 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea29fe6dd4c424c80c6026f878676ea" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" is_pre_election: true
I20240501 08:10:41.511680 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea29fe6dd4c424c80c6026f878676ea" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.511914 25265 raft_consensus.cc:2454] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.511915 25189 raft_consensus.cc:2454] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.512344 25304 leader_election.cc:302] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.512835 25392 raft_consensus.cc:2781] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.512967 25392 raft_consensus.cc:482] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.513089 25392 raft_consensus.cc:3037] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.515246 25392 raft_consensus.cc:504] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.515815 25392 leader_election.cc:288] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Requested vote from peers 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569), 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463)
I20240501 08:10:41.516232 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea29fe6dd4c424c80c6026f878676ea" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
I20240501 08:10:41.516254 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea29fe6dd4c424c80c6026f878676ea" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.516428 25265 raft_consensus.cc:3037] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.516435 25189 raft_consensus.cc:3037] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.518543 25265 raft_consensus.cc:2454] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.518971 25305 leader_election.cc:302] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.519063 25189 raft_consensus.cc:2454] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.519300 25392 raft_consensus.cc:2781] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.519587 25392 raft_consensus.cc:686] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Becoming Leader. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Running, Role: LEADER
I20240501 08:10:41.519888 25392 consensus_queue.cc:237] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.522161 25088 catalog_manager.cc:5484] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3). New cstate: current_term: 1 leader_uuid: "8afbcfee573c49a7a3185af453c8b0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:41.530068 25420 raft_consensus.cc:482] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.530197 25420 raft_consensus.cc:504] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.530720 25420 leader_election.cc:288] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.531121 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3437ee7ff472caffbc61fa5f5fb38" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe" is_pre_election: true
I20240501 08:10:41.531113 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3437ee7ff472caffbc61fa5f5fb38" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.531361 25339 raft_consensus.cc:2454] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.531379 25189 raft_consensus.cc:2454] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 0.
I20240501 08:10:41.531764 25231 leader_election.cc:302] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.531996 25420 raft_consensus.cc:2781] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.532110 25420 raft_consensus.cc:482] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.532199 25420 raft_consensus.cc:3037] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.534034 25420 raft_consensus.cc:504] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.534518 25420 leader_election.cc:288] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:41.534883 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3437ee7ff472caffbc61fa5f5fb38" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.534941 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d3437ee7ff472caffbc61fa5f5fb38" candidate_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:41.535056 25189 raft_consensus.cc:3037] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.535151 25339 raft_consensus.cc:3037] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.537072 25339 raft_consensus.cc:2454] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.537070 25189 raft_consensus.cc:2454] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809abcd9cb7a409d8ddddbc3e0dd2dd3 in term 1.
I20240501 08:10:41.537431 25231 leader_election.cc:302] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.537690 25420 raft_consensus.cc:2781] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.537830 25420 raft_consensus.cc:686] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Running, Role: LEADER
I20240501 08:10:41.538120 25420 consensus_queue.cc:237] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.540568 25088 catalog_manager.cc:5484] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:41.572573 25392 raft_consensus.cc:482] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:41.572811 25392 raft_consensus.cc:504] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.573441 25392 leader_election.cc:288] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569)
I20240501 08:10:41.573957 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343e4dfc3c484385a9eb31ee16c5431f" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" is_pre_election: true
I20240501 08:10:41.573980 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343e4dfc3c484385a9eb31ee16c5431f" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" is_pre_election: true
I20240501 08:10:41.574170 25189 raft_consensus.cc:2454] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.574173 25265 raft_consensus.cc:2454] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 0.
I20240501 08:10:41.574512 25304 leader_election.cc:302] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8afbcfee573c49a7a3185af453c8b0fe, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:41.574752 25392 raft_consensus.cc:2781] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:41.574877 25392 raft_consensus.cc:482] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:41.575434 25392 raft_consensus.cc:3037] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.577363 25392 raft_consensus.cc:504] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.577869 25392 leader_election.cc:288] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Requested vote from peers 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1:42463), 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569)
I20240501 08:10:41.578316 25189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343e4dfc3c484385a9eb31ee16c5431f" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
I20240501 08:10:41.578323 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343e4dfc3c484385a9eb31ee16c5431f" candidate_uuid: "8afbcfee573c49a7a3185af453c8b0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
I20240501 08:10:41.578505 25189 raft_consensus.cc:3037] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.578512 25265 raft_consensus.cc:3037] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.580430 25265 raft_consensus.cc:2454] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.580466 25189 raft_consensus.cc:2454] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8afbcfee573c49a7a3185af453c8b0fe in term 1.
I20240501 08:10:41.580773 25305 leader_election.cc:302] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809abcd9cb7a409d8ddddbc3e0dd2dd3, 8afbcfee573c49a7a3185af453c8b0fe; no voters:
I20240501 08:10:41.580997 25392 raft_consensus.cc:2781] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.581141 25392 raft_consensus.cc:686] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Becoming Leader. State: Replica: 8afbcfee573c49a7a3185af453c8b0fe, State: Running, Role: LEADER
I20240501 08:10:41.581521 25392 consensus_queue.cc:237] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:41.584091 25088 catalog_manager.cc:5484] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3). New cstate: current_term: 1 leader_uuid: "8afbcfee573c49a7a3185af453c8b0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:41.696116 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.698907 25434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.699780 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.700004 25437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.700814 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:41.701063 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:41.701134 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:41.701176 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551041701162 us; error 0 us; skew 500 ppm
I20240501 08:10:41.701308 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.702541 24528 webserver.cc:460] Webserver started at http://127.23.244.4:39181/ using document root <none> and password file <none>
I20240501 08:10:41.702777 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.702849 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.702967 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.703594 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-3-root/instance:
uuid: "21d1f94bf33342feb097a7981acb68fb"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-fck4"
I20240501 08:10:41.706506 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:41.707996 25442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.708366 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:41.708494 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-3-root
uuid: "21d1f94bf33342feb097a7981acb68fb"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-fck4"
I20240501 08:10:41.708612 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714551036644763-24528-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.732232 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.733346 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.733929 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.734947 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.735057 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.735167 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.735235 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.753593 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.4:42469
I20240501 08:10:41.753638 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.4:42469 every 8 connection(s)
I20240501 08:10:41.761104 25505 heartbeater.cc:347] Connected to a master server at 127.23.244.62:33625
I20240501 08:10:41.761237 25505 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.761672 25505 heartbeater.cc:511] Master 127.23.244.62:33625 requested a full tablet report, sending...
I20240501 08:10:41.762496 25088 ts_manager.cc:194] Registered new tserver with Master: 21d1f94bf33342feb097a7981acb68fb (127.23.244.4:42469)
I20240501 08:10:41.763235 25088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36796
I20240501 08:10:41.767776 25420 consensus_queue.cc:1035] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.771878 25393 consensus_queue.cc:1035] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.802306 25393 consensus_queue.cc:1035] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.805989 25393 consensus_queue.cc:1035] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.847754 25390 consensus_queue.cc:1035] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.851585 25425 consensus_queue.cc:1035] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.852761 25393 consensus_queue.cc:1035] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.857277 25419 consensus_queue.cc:1035] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.869411 25141 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:41.877696 25265 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34"
dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
mode: GRACEFUL
new_leader_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
from {username='slave'} at 127.0.0.1:56562
I20240501 08:10:41.878003 25265 raft_consensus.cc:595] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Received request to transfer leadership to TS 95a81083b5e4442eb7c325dfcd8ee34d
I20240501 08:10:41.878576 25141 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:10:41.878839 25141 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:41.950829 25392 consensus_queue.cc:1035] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:41.954629 25392 consensus_queue.cc:1035] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.032223 25419 consensus_queue.cc:1035] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.036700 25419 consensus_queue.cc:1035] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.065436 25392 consensus_queue.cc:1035] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.069398 25522 consensus_queue.cc:1035] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.108160 25522 consensus_queue.cc:1035] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.112012 25522 consensus_queue.cc:1035] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:42.331692 25420 raft_consensus.cc:979] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: : Instructing follower 95a81083b5e4442eb7c325dfcd8ee34d to start an election
I20240501 08:10:42.331859 25420 raft_consensus.cc:1067] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Signalling peer 95a81083b5e4442eb7c325dfcd8ee34d to start an election
I20240501 08:10:42.332661 25189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34"
dest_uuid: "95a81083b5e4442eb7c325dfcd8ee34d"
from {username='slave'} at 127.0.0.1:35786
I20240501 08:10:42.332901 25189 raft_consensus.cc:482] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:10:42.333014 25189 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:10:42.335458 25189 raft_consensus.cc:504] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:42.336028 25189 leader_election.cc:288] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 2 election: Requested vote from peers 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2:43569), 8afbcfee573c49a7a3185af453c8b0fe (127.23.244.3:33535)
I20240501 08:10:42.336495 25265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3"
I20240501 08:10:42.336548 25339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e1ae299e884a299ad04395af4ffb34" candidate_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8afbcfee573c49a7a3185af453c8b0fe"
I20240501 08:10:42.336678 25265 raft_consensus.cc:3032] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:10:42.336795 25339 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:10:42.336823 25265 raft_consensus.cc:729] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809abcd9cb7a409d8ddddbc3e0dd2dd3, State: Running, Role: LEADER
I20240501 08:10:42.337116 25265 consensus_queue.cc:260] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:42.337469 25265 raft_consensus.cc:3037] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:10:42.338780 25339 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 2.
I20240501 08:10:42.339148 25156 leader_election.cc:302] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8afbcfee573c49a7a3185af453c8b0fe, 95a81083b5e4442eb7c325dfcd8ee34d; no voters:
I20240501 08:10:42.339406 25530 raft_consensus.cc:2781] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:10:42.339545 25530 raft_consensus.cc:686] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 2 LEADER]: Becoming Leader. State: Replica: 95a81083b5e4442eb7c325dfcd8ee34d, State: Running, Role: LEADER
I20240501 08:10:42.339516 25265 raft_consensus.cc:2454] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a81083b5e4442eb7c325dfcd8ee34d in term 2.
I20240501 08:10:42.339838 25530 consensus_queue.cc:237] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } }
I20240501 08:10:42.342065 25088 catalog_manager.cc:5484] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d reported cstate change: term changed from 1 to 2, leader changed from 809abcd9cb7a409d8ddddbc3e0dd2dd3 (127.23.244.2) to 95a81083b5e4442eb7c325dfcd8ee34d (127.23.244.1). New cstate: current_term: 2 leader_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:42.764410 25505 heartbeater.cc:503] Master 127.23.244.62:33625 was elected leader, sending a full tablet report...
I20240501 08:10:42.798555 25339 raft_consensus.cc:1261] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 2 FOLLOWER]: Refusing update from remote peer 95a81083b5e4442eb7c325dfcd8ee34d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:10:42.799221 25530 consensus_queue.cc:1035] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:42.802554 25265 raft_consensus.cc:1261] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 2 FOLLOWER]: Refusing update from remote peer 95a81083b5e4442eb7c325dfcd8ee34d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:10:42.803045 25425 consensus_queue.cc:1035] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:43.879209 25141 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:43.880456 25141 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:10:43.880589 25141 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:45.881037 25141 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:45.882467 25141 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:10:45.882635 25141 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:46.881392 25338 consensus_queue.cc:237] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } }
I20240501 08:10:46.885238 25265 raft_consensus.cc:1261] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Refusing update from remote peer 8afbcfee573c49a7a3185af453c8b0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.885949 25542 consensus_queue.cc:1035] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.886067 25190 raft_consensus.cc:1261] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Refusing update from remote peer 8afbcfee573c49a7a3185af453c8b0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.886606 25542 consensus_queue.cc:1035] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.889299 25522 raft_consensus.cc:2932] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.891022 25265 raft_consensus.cc:2932] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.892503 25190 raft_consensus.cc:2932] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
W20240501 08:10:46.893257 25302 consensus_peers.cc:485] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe -> Peer 21d1f94bf33342feb097a7981acb68fb (127.23.244.4:42469): Couldn't send request to peer 21d1f94bf33342feb097a7981acb68fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ea29fe6dd4c424c80c6026f878676ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:46.895480 25086 catalog_manager.cc:5484] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "8afbcfee573c49a7a3185af453c8b0fe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } attrs { replace: true } } peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.905913 25265 consensus_queue.cc:237] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } }
I20240501 08:10:46.908476 25338 raft_consensus.cc:1261] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Refusing update from remote peer 809abcd9cb7a409d8ddddbc3e0dd2dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.908560 25190 raft_consensus.cc:1261] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Refusing update from remote peer 809abcd9cb7a409d8ddddbc3e0dd2dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.909273 25420 consensus_queue.cc:1035] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.910135 25420 consensus_queue.cc:1035] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.913956 25420 raft_consensus.cc:2932] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.914436 25190 raft_consensus.cc:2932] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.920459 25087 catalog_manager.cc:5484] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:46.922699 25265 consensus_queue.cc:237] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } }
I20240501 08:10:46.927042 25190 raft_consensus.cc:1261] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Refusing update from remote peer 809abcd9cb7a409d8ddddbc3e0dd2dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.927459 25339 raft_consensus.cc:1261] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Refusing update from remote peer 809abcd9cb7a409d8ddddbc3e0dd2dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.928367 25554 consensus_queue.cc:1035] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.929214 25555 consensus_queue.cc:1035] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:10:46.931483 25228 consensus_peers.cc:485] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 -> Peer 21d1f94bf33342feb097a7981acb68fb (127.23.244.4:42469): Couldn't send request to peer 21d1f94bf33342feb097a7981acb68fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec175c6ee540428a987372f7e5f629f1. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:10:46.931697 25228 consensus_peers.cc:485] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 -> Peer 21d1f94bf33342feb097a7981acb68fb (127.23.244.4:42469): Couldn't send request to peer 21d1f94bf33342feb097a7981acb68fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d3437ee7ff472caffbc61fa5f5fb38. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:46.932487 25554 raft_consensus.cc:2932] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.934156 25190 raft_consensus.cc:2932] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.938187 25088 catalog_manager.cc:5484] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 reported cstate change: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:46.940064 25337 consensus_queue.cc:237] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } }
I20240501 08:10:46.943890 25265 raft_consensus.cc:1261] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Refusing update from remote peer 8afbcfee573c49a7a3185af453c8b0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.943899 25338 raft_consensus.cc:2932] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } attrs { replace: true } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.945135 25522 consensus_queue.cc:1035] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:10:46.945847 25190 raft_consensus.cc:1261] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Refusing update from remote peer 8afbcfee573c49a7a3185af453c8b0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:46.946385 25522 consensus_queue.cc:1035] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:10:46.947372 25302 consensus_peers.cc:485] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe -> Peer 21d1f94bf33342feb097a7981acb68fb (127.23.244.4:42469): Couldn't send request to peer 21d1f94bf33342feb097a7981acb68fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bede93e312574be3927b1e35d259ca57. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:46.951308 25337 raft_consensus.cc:2932] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.954224 25543 raft_consensus.cc:2932] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.954860 25265 raft_consensus.cc:2932] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:46.961009 25088 catalog_manager.cc:5484] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe reported cstate change: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "8afbcfee573c49a7a3185af453c8b0fe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:46.968976 25189 raft_consensus.cc:2932] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 21d1f94bf33342feb097a7981acb68fb (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "95a81083b5e4442eb7c325dfcd8ee34d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42463 } attrs { replace: true } } peers { permanent_uuid: "809abcd9cb7a409d8ddddbc3e0dd2dd3" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 43569 } } peers { permanent_uuid: "8afbcfee573c49a7a3185af453c8b0fe" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 33535 } } peers { permanent_uuid: "21d1f94bf33342feb097a7981acb68fb" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 42469 } attrs { promote: true } } }
I20240501 08:10:47.012488 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:10:47.021471 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:47.021734 24528 tablet_replica.cc:330] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.021972 24528 raft_consensus.cc:2229] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.022183 24528 raft_consensus.cc:2258] T 343e4dfc3c484385a9eb31ee16c5431f P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.022840 24528 tablet_replica.cc:330] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.023041 24528 raft_consensus.cc:2229] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.023322 24528 raft_consensus.cc:2258] T b6d3437ee7ff472caffbc61fa5f5fb38 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.024017 24528 tablet_replica.cc:330] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.024180 24528 raft_consensus.cc:2229] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.024399 24528 raft_consensus.cc:2258] T ec175c6ee540428a987372f7e5f629f1 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.025027 24528 tablet_replica.cc:330] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.025255 24528 raft_consensus.cc:2229] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.025396 24528 raft_consensus.cc:2258] T 0b83f73f70e24f3ca20628cda62e8925 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.025890 24528 tablet_replica.cc:330] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.026005 24528 raft_consensus.cc:2229] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.026346 24528 raft_consensus.cc:2258] T f5795a466f3c4c938eb00d01fe3c0ed2 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.026782 24528 tablet_replica.cc:330] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.026961 24528 raft_consensus.cc:2229] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.027263 24528 raft_consensus.cc:2258] T b2e1ae299e884a299ad04395af4ffb34 P 95a81083b5e4442eb7c325dfcd8ee34d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.027731 24528 tablet_replica.cc:330] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.027881 24528 raft_consensus.cc:2229] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.028118 24528 raft_consensus.cc:2258] T 7ea29fe6dd4c424c80c6026f878676ea P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.028831 24528 tablet_replica.cc:330] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d: stopping tablet replica
I20240501 08:10:47.028995 24528 raft_consensus.cc:2229] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.029186 24528 raft_consensus.cc:2258] T bede93e312574be3927b1e35d259ca57 P 95a81083b5e4442eb7c325dfcd8ee34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.045469 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:10:47.050621 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
W20240501 08:10:47.052469 25075 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240501 08:10:47.054487 25136 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.054579 25136 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.058382 25136 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.058463 25136 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:10:47.060878 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:47.061233 24528 tablet_replica.cc:330] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.061461 24528 raft_consensus.cc:2229] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.061851 24528 raft_consensus.cc:2258] T ec175c6ee540428a987372f7e5f629f1 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.062651 24528 tablet_replica.cc:330] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.062856 24528 raft_consensus.cc:2229] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.063167 24528 raft_consensus.cc:2258] T 0b83f73f70e24f3ca20628cda62e8925 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.063740 24528 tablet_replica.cc:330] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.063938 24528 raft_consensus.cc:2229] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.064131 24528 raft_consensus.cc:2258] T 7ea29fe6dd4c424c80c6026f878676ea P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.064918 24528 tablet_replica.cc:330] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.065097 24528 raft_consensus.cc:2229] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.065326 24528 raft_consensus.cc:2258] T bede93e312574be3927b1e35d259ca57 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.066006 24528 tablet_replica.cc:330] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.066179 24528 raft_consensus.cc:2229] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.066326 24528 raft_consensus.cc:2258] T f5795a466f3c4c938eb00d01fe3c0ed2 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.066743 24528 tablet_replica.cc:330] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.066897 24528 raft_consensus.cc:2229] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.067107 24528 raft_consensus.cc:2258] T b2e1ae299e884a299ad04395af4ffb34 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.067516 24528 tablet_replica.cc:330] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.067670 24528 raft_consensus.cc:2229] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.067802 24528 raft_consensus.cc:2258] T 343e4dfc3c484385a9eb31ee16c5431f P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.068368 24528 tablet_replica.cc:330] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3: stopping tablet replica
I20240501 08:10:47.068513 24528 raft_consensus.cc:2229] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.068912 24528 raft_consensus.cc:2258] T b6d3437ee7ff472caffbc61fa5f5fb38 P 809abcd9cb7a409d8ddddbc3e0dd2dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.075373 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:10:47.079671 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
W20240501 08:10:47.082921 25136 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.083037 25136 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.085549 25136 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 08:10:47.085706 25136 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 08:10:47.089170 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:47.089422 24528 tablet_replica.cc:330] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.089581 24528 raft_consensus.cc:2229] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.089977 24528 raft_consensus.cc:2258] T bede93e312574be3927b1e35d259ca57 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.090638 24528 tablet_replica.cc:330] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.090806 24528 raft_consensus.cc:2229] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.091104 24528 raft_consensus.cc:2258] T 7ea29fe6dd4c424c80c6026f878676ea P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.091639 24528 tablet_replica.cc:330] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.091773 24528 raft_consensus.cc:2229] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.091964 24528 raft_consensus.cc:2258] T b6d3437ee7ff472caffbc61fa5f5fb38 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.092530 24528 tablet_replica.cc:330] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.092803 24528 raft_consensus.cc:2229] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.092970 24528 raft_consensus.cc:2258] T b2e1ae299e884a299ad04395af4ffb34 P 8afbcfee573c49a7a3185af453c8b0fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.093400 24528 tablet_replica.cc:330] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.093534 24528 raft_consensus.cc:2229] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.093676 24528 raft_consensus.cc:2258] T f5795a466f3c4c938eb00d01fe3c0ed2 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.094066 24528 tablet_replica.cc:330] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.094223 24528 raft_consensus.cc:2229] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.094379 24528 raft_consensus.cc:2258] T ec175c6ee540428a987372f7e5f629f1 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.094877 24528 tablet_replica.cc:330] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.095032 24528 raft_consensus.cc:2229] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.095273 24528 raft_consensus.cc:2258] T 343e4dfc3c484385a9eb31ee16c5431f P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.095695 24528 tablet_replica.cc:330] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe: stopping tablet replica
I20240501 08:10:47.095865 24528 raft_consensus.cc:2229] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:47.095999 24528 raft_consensus.cc:2258] T 0b83f73f70e24f3ca20628cda62e8925 P 8afbcfee573c49a7a3185af453c8b0fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.101286 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:10:47.104910 24528 tablet_server.cc:178] TabletServer@127.23.244.4:0 shutting down...
I20240501 08:10:47.112901 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:47.125471 24528 tablet_server.cc:194] TabletServer@127.23.244.4:0 shutdown complete.
I20240501 08:10:47.127954 24528 master.cc:537] Master@127.23.244.62:33625 shutting down...
I20240501 08:10:47.135010 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:47.135203 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:47.135295 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 43287369b0614e0f93647b54f666c9ce: stopping tablet replica
I20240501 08:10:47.149372 24528 master.cc:552] Master@127.23.244.62:33625 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7367 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 08:10:47.165714 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:45949,127.23.244.61:41719,127.23.244.60:33137
I20240501 08:10:47.166301 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:47.168792 25566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.169355 25567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.169955 25569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.170454 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:47.170684 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.170740 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.170786 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047170764 us; error 0 us; skew 500 ppm
I20240501 08:10:47.170898 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.172030 24528 webserver.cc:460] Webserver started at http://127.23.244.62:43735/ using document root <none> and password file <none>
I20240501 08:10:47.172217 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.172297 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.172432 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.173240 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "82143f30768b4be3959cc080edd7d282"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.176223 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:47.177723 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.178064 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:47.178169 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "82143f30768b4be3959cc080edd7d282"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.178299 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.199007 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.199589 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.211949 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:45949
I20240501 08:10:47.211983 25625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:45949 every 8 connection(s)
I20240501 08:10:47.213230 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.213281 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:47.216519 25626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:47.216640 25629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.216897 25628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.217685 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:47.218488 25631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.218896 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.218964 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.219030 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047218986 us; error 0 us; skew 500 ppm
I20240501 08:10:47.219214 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.220588 24528 webserver.cc:460] Webserver started at http://127.23.244.61:42185/ using document root <none> and password file <none>
I20240501 08:10:47.220840 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.220942 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.221073 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.221698 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root/instance:
uuid: "67a64a700f5c419a889985c431c7cb7b"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.223703 25626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:47.224354 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
W20240501 08:10:47.225638 25626 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.61:41719: Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:47.226110 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.226487 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:47.226577 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
uuid: "67a64a700f5c419a889985c431c7cb7b"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.226677 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.240350 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.240918 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.252506 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.61:41719
I20240501 08:10:47.252534 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.61:41719 every 8 connection(s)
I20240501 08:10:47.253796 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:47.253917 25691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:10:47.256439 25693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.257002 25694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.257551 25691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:47.257755 25696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.258687 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:47.258957 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.259023 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.259099 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047259062 us; error 0 us; skew 500 ppm
I20240501 08:10:47.259276 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.260587 24528 webserver.cc:460] Webserver started at http://127.23.244.60:46757/ using document root <none> and password file <none>
I20240501 08:10:47.260846 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.260955 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.261090 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.261749 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root/instance:
uuid: "8e6b24f64f234cdd9c86f7e2d3d20629"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.264112 25691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:47.264441 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:10:47.266499 25703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.266884 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:47.267030 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
uuid: "8e6b24f64f234cdd9c86f7e2d3d20629"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.267158 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.271206 25691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:47.272990 25691 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.60:33137: Network error: Client connection negotiation failed: client connection to 127.23.244.60:33137: connect: Connection refused (error 111)
I20240501 08:10:47.283728 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.284322 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.287935 25626 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } attempt: 1
I20240501 08:10:47.295187 25626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
W20240501 08:10:47.297214 25626 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.244.60:33137: Network error: Client connection negotiation failed: client connection to 127.23.244.60:33137: connect: Connection refused (error 111)
I20240501 08:10:47.299923 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.60:33137
I20240501 08:10:47.299962 25755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.60:33137 every 8 connection(s)
I20240501 08:10:47.300827 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:10:47.301265 25756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.303463 25756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:47.305191 25691 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } attempt: 1
I20240501 08:10:47.310561 25756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:47.315464 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: Bootstrap starting.
I20240501 08:10:47.316665 25756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
I20240501 08:10:47.317063 25691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.318953 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: No bootstrap required, opened a new log
I20240501 08:10:47.319600 25691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.320199 25691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.320281 25691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a64a700f5c419a889985c431c7cb7b, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.320547 25691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.321220 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.321588 25691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:47.321588 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.325166 25756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: Bootstrap starting.
I20240501 08:10:47.326704 25756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: Neither blocks nor log segments found. Creating new log.
W20240501 08:10:47.328349 25772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:47.328449 25772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:10:47.329175 25756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: No bootstrap required, opened a new log
I20240501 08:10:47.329725 25756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.329887 25756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.329942 25756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6b24f64f234cdd9c86f7e2d3d20629, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.330155 25756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.330703 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.330886 25756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:47.330893 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.333447 25626 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } attempt: 1
W20240501 08:10:47.336990 25785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:47.337096 25785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:10:47.341787 25626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: Bootstrap starting.
I20240501 08:10:47.343041 25626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.344612 25626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: No bootstrap required, opened a new log
I20240501 08:10:47.345211 25626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.345388 25626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.345464 25626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82143f30768b4be3959cc080edd7d282, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.345695 25626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.346309 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.346499 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.346508 25626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:47.350445 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 08:10:47.350548 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 08:10:47.351241 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:47.351603 25798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:10:47.351684 25798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 08:10:47.354449 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.355747 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.356052 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.356806 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:47.357045 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.357124 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.357185 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047357167 us; error 0 us; skew 500 ppm
I20240501 08:10:47.357352 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.358623 24528 webserver.cc:460] Webserver started at http://127.23.244.1:33787/ using document root <none> and password file <none>
I20240501 08:10:47.358862 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.358942 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.359552 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.360186 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "56cd2ae440014564886a29d2e1d8501a"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.362650 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240501 08:10:47.364253 25807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.364593 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:10:47.364696 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "56cd2ae440014564886a29d2e1d8501a"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.364853 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.373519 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.374091 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.374655 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:47.375521 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:47.375608 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.375705 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:47.375783 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.392956 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:39405
I20240501 08:10:47.392994 25869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:39405 every 8 connection(s)
I20240501 08:10:47.403507 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:47.403754 25870 heartbeater.cc:347] Connected to a master server at 127.23.244.62:45949
I20240501 08:10:47.403928 25870 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.404475 25870 heartbeater.cc:511] Master 127.23.244.62:45949 requested a full tablet report, sending...
I20240501 08:10:47.404839 25871 heartbeater.cc:347] Connected to a master server at 127.23.244.60:33137
I20240501 08:10:47.404989 25871 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.405540 25871 heartbeater.cc:511] Master 127.23.244.60:33137 requested a full tablet report, sending...
I20240501 08:10:47.405726 25591 ts_manager.cc:194] Registered new tserver with Master: 56cd2ae440014564886a29d2e1d8501a (127.23.244.1:39405)
I20240501 08:10:47.405866 25872 heartbeater.cc:347] Connected to a master server at 127.23.244.61:41719
I20240501 08:10:47.405987 25872 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.406627 25872 heartbeater.cc:511] Master 127.23.244.61:41719 requested a full tablet report, sending...
I20240501 08:10:47.406886 25721 ts_manager.cc:194] Registered new tserver with Master: 56cd2ae440014564886a29d2e1d8501a (127.23.244.1:39405)
I20240501 08:10:47.408249 25656 ts_manager.cc:194] Registered new tserver with Master: 56cd2ae440014564886a29d2e1d8501a (127.23.244.1:39405)
W20240501 08:10:47.408811 25877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.409922 25878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.410599 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:47.411437 25880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.411787 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.411855 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.411928 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047411893 us; error 0 us; skew 500 ppm
I20240501 08:10:47.412070 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.413390 24528 webserver.cc:460] Webserver started at http://127.23.244.2:45995/ using document root <none> and password file <none>
I20240501 08:10:47.413597 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.413694 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.413820 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.414460 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "315d6e5186064f7290334850c1bcb97a"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.416844 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20240501 08:10:47.418388 25885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.418794 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:47.418884 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "315d6e5186064f7290334850c1bcb97a"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.418994 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.426071 25761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:47.426241 25761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.427017 25761 leader_election.cc:288] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 8e6b24f64f234cdd9c86f7e2d3d20629 (127.23.244.60:33137)
I20240501 08:10:47.427498 25731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a64a700f5c419a889985c431c7cb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" is_pre_election: true
I20240501 08:10:47.427520 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a64a700f5c419a889985c431c7cb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282" is_pre_election: true
I20240501 08:10:47.427786 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a64a700f5c419a889985c431c7cb7b in term 0.
I20240501 08:10:47.427790 25731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a64a700f5c419a889985c431c7cb7b in term 0.
I20240501 08:10:47.428287 25641 leader_election.cc:302] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a64a700f5c419a889985c431c7cb7b, 82143f30768b4be3959cc080edd7d282; no voters:
I20240501 08:10:47.428536 25761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:47.428653 25761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:47.428764 25761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.430349 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.431048 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.431710 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:47.432194 25761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.432895 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:47.432986 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.433095 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:47.433169 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.433497 25731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a64a700f5c419a889985c431c7cb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629"
I20240501 08:10:47.433656 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a64a700f5c419a889985c431c7cb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282"
I20240501 08:10:47.433871 25601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.436905 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a64a700f5c419a889985c431c7cb7b in term 1.
I20240501 08:10:47.433733 25731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.437460 25641 leader_election.cc:302] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a64a700f5c419a889985c431c7cb7b, 82143f30768b4be3959cc080edd7d282; no voters:
I20240501 08:10:47.438103 25897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:47.438668 25897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 1 LEADER]: Becoming Leader. State: Replica: 67a64a700f5c419a889985c431c7cb7b, State: Running, Role: LEADER
I20240501 08:10:47.439036 25897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:47.440408 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a64a700f5c419a889985c431c7cb7b. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.440698 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:47.441013 25731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a64a700f5c419a889985c431c7cb7b in term 1.
I20240501 08:10:47.441308 25761 leader_election.cc:288] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [CANDIDATE]: Term 1 election: Requested vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 8e6b24f64f234cdd9c86f7e2d3d20629 (127.23.244.60:33137)
I20240501 08:10:47.444242 25901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:47.446615 25901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:47.451342 25601 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Refusing update from remote peer 67a64a700f5c419a889985c431c7cb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:47.451454 25731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Refusing update from remote peer 67a64a700f5c419a889985c431c7cb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:47.452065 25761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.452359 25899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.461494 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a64a700f5c419a889985c431c7cb7b. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.461865 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.466585 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.467799 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:47.468868 25901 catalog_manager.cc:1347] Generated new cluster ID: cfe10519ca1948e3b7c06a382e311a95
I20240501 08:10:47.468998 25901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:47.468974 25774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.469285 25774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.473322 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a64a700f5c419a889985c431c7cb7b. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.473649 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.475406 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.475672 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:47.478027 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67a64a700f5c419a889985c431c7cb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:47.478502 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 08:10:47.485898 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:46257 every 8 connection(s)
I20240501 08:10:47.485895 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:46257
I20240501 08:10:47.496186 25901 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:47.496385 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:47.497390 25901 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:47.517657 25960 heartbeater.cc:347] Connected to a master server at 127.23.244.62:45949
I20240501 08:10:47.518242 25960 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.518842 25960 heartbeater.cc:511] Master 127.23.244.62:45949 requested a full tablet report, sending...
I20240501 08:10:47.519407 25961 heartbeater.cc:347] Connected to a master server at 127.23.244.60:33137
I20240501 08:10:47.519547 25961 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.520064 25961 heartbeater.cc:511] Master 127.23.244.60:33137 requested a full tablet report, sending...
I20240501 08:10:47.520220 25591 ts_manager.cc:194] Registered new tserver with Master: 315d6e5186064f7290334850c1bcb97a (127.23.244.2:46257)
I20240501 08:10:47.520987 25721 ts_manager.cc:194] Registered new tserver with Master: 315d6e5186064f7290334850c1bcb97a (127.23.244.2:46257)
I20240501 08:10:47.522694 25901 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: Generated new TSK 0
I20240501 08:10:47.523077 25901 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 08:10:47.527182 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.528527 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.529747 25962 heartbeater.cc:347] Connected to a master server at 127.23.244.61:41719
I20240501 08:10:47.529871 25962 heartbeater.cc:464] Registering TS with master...
W20240501 08:10:47.530030 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.530331 25962 heartbeater.cc:511] Master 127.23.244.61:41719 requested a full tablet report, sending...
I20240501 08:10:47.531294 25656 ts_manager.cc:194] Registered new tserver with Master: 315d6e5186064f7290334850c1bcb97a (127.23.244.2:46257)
I20240501 08:10:47.533216 25656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42858
I20240501 08:10:47.534170 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:47.534422 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:47.534495 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:47.534554 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551047534536 us; error 0 us; skew 500 ppm
I20240501 08:10:47.534708 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.536581 24528 webserver.cc:460] Webserver started at http://127.23.244.3:41945/ using document root <none> and password file <none>
I20240501 08:10:47.536831 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.536909 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.537046 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.537638 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "61317ad828ea4ca68919d61a4dcaa008"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.539870 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:10:47.541314 25975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.541658 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:10:47.541771 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "61317ad828ea4ca68919d61a4dcaa008"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-fck4"
I20240501 08:10:47.541883 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.563661 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.564639 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.565227 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:47.566144 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:47.566226 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.566324 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:47.566397 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.584872 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:42773
I20240501 08:10:47.584908 26037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:42773 every 8 connection(s)
I20240501 08:10:47.605774 26039 heartbeater.cc:347] Connected to a master server at 127.23.244.60:33137
I20240501 08:10:47.605975 26039 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.606082 26038 heartbeater.cc:347] Connected to a master server at 127.23.244.62:45949
I20240501 08:10:47.606199 26038 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.606446 26039 heartbeater.cc:511] Master 127.23.244.60:33137 requested a full tablet report, sending...
I20240501 08:10:47.606572 26038 heartbeater.cc:511] Master 127.23.244.62:45949 requested a full tablet report, sending...
I20240501 08:10:47.607396 26040 heartbeater.cc:347] Connected to a master server at 127.23.244.61:41719
I20240501 08:10:47.607427 25721 ts_manager.cc:194] Registered new tserver with Master: 61317ad828ea4ca68919d61a4dcaa008 (127.23.244.3:42773)
I20240501 08:10:47.607606 26040 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.607678 25591 ts_manager.cc:194] Registered new tserver with Master: 61317ad828ea4ca68919d61a4dcaa008 (127.23.244.3:42773)
I20240501 08:10:47.608134 26040 heartbeater.cc:511] Master 127.23.244.61:41719 requested a full tablet report, sending...
I20240501 08:10:47.608891 25656 ts_manager.cc:194] Registered new tserver with Master: 61317ad828ea4ca68919d61a4dcaa008 (127.23.244.3:42773)
I20240501 08:10:47.609187 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016165404s
I20240501 08:10:47.609913 25656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42862
I20240501 08:10:48.338339 25785 catalog_manager.cc:1259] Loaded cluster ID: cfe10519ca1948e3b7c06a382e311a95
I20240501 08:10:48.338445 25785 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: loading cluster ID for follower catalog manager: success
I20240501 08:10:48.340333 25785 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: acquiring CA information for follower catalog manager: success
I20240501 08:10:48.341321 25785 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:10:48.352798 25798 catalog_manager.cc:1259] Loaded cluster ID: cfe10519ca1948e3b7c06a382e311a95
I20240501 08:10:48.352885 25798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: loading cluster ID for follower catalog manager: success
I20240501 08:10:48.354810 25798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: acquiring CA information for follower catalog manager: success
I20240501 08:10:48.356077 25798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 08:10:48.410219 25656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42842
I20240501 08:10:48.534425 25962 heartbeater.cc:503] Master 127.23.244.61:41719 was elected leader, sending a full tablet report...
I20240501 08:10:48.611071 26040 heartbeater.cc:503] Master 127.23.244.61:41719 was elected leader, sending a full tablet report...
I20240501 08:10:49.326479 25771 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:49.411298 25872 heartbeater.cc:503] Master 127.23.244.61:41719 was elected leader, sending a full tablet report...
I20240501 08:10:49.667984 24528 test_util.cc:268] Using random seed: 105104738
I20240501 08:10:49.700656 25656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42876:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:10:49.732621 26003 tablet_service.cc:1467] Processing CreateTablet for tablet 65fc47acc3434b4185a17f49b16bd3ac (DEFAULT_TABLE table=test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:10:49.733403 26003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fc47acc3434b4185a17f49b16bd3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.733403 25835 tablet_service.cc:1467] Processing CreateTablet for tablet 9d065795ef514a36948b4b7ebcab3f3c (DEFAULT_TABLE table=test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:10:49.733825 25835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d065795ef514a36948b4b7ebcab3f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.737406 25918 tablet_service.cc:1467] Processing CreateTablet for tablet 376f575201ce4b2aa874d89ee2153396 (DEFAULT_TABLE table=test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:10:49.737803 25918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376f575201ce4b2aa874d89ee2153396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.741936 26074 tablet_bootstrap.cc:492] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: Bootstrap starting.
I20240501 08:10:49.744621 26075 tablet_bootstrap.cc:492] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: Bootstrap starting.
I20240501 08:10:49.744843 26074 tablet_bootstrap.cc:654] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.744988 26073 tablet_bootstrap.cc:492] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: Bootstrap starting.
I20240501 08:10:49.746835 26075 tablet_bootstrap.cc:654] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.747140 26073 tablet_bootstrap.cc:654] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.747308 26074 tablet_bootstrap.cc:492] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: No bootstrap required, opened a new log
I20240501 08:10:49.747470 26074 ts_tablet_manager.cc:1367] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 08:10:49.748848 26074 raft_consensus.cc:348] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61317ad828ea4ca68919d61a4dcaa008" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 42773 } }
I20240501 08:10:49.749089 26074 raft_consensus.cc:374] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.749207 26074 raft_consensus.cc:729] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61317ad828ea4ca68919d61a4dcaa008, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.749470 26074 consensus_queue.cc:260] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61317ad828ea4ca68919d61a4dcaa008" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 42773 } }
I20240501 08:10:49.749622 26075 tablet_bootstrap.cc:492] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: No bootstrap required, opened a new log
I20240501 08:10:49.749658 26073 tablet_bootstrap.cc:492] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: No bootstrap required, opened a new log
I20240501 08:10:49.749679 26074 raft_consensus.cc:388] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:49.749764 26075 ts_tablet_manager.cc:1367] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:49.749825 26074 raft_consensus.cc:482] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:49.749881 26073 ts_tablet_manager.cc:1367] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:49.749986 26074 raft_consensus.cc:3037] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.750635 26075 raft_consensus.cc:348] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315d6e5186064f7290334850c1bcb97a" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46257 } }
I20240501 08:10:49.750689 26073 raft_consensus.cc:348] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd2ae440014564886a29d2e1d8501a" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 39405 } }
I20240501 08:10:49.750799 26075 raft_consensus.cc:374] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.750874 26073 raft_consensus.cc:374] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.751408 26075 raft_consensus.cc:729] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d6e5186064f7290334850c1bcb97a, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.751435 26073 raft_consensus.cc:729] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd2ae440014564886a29d2e1d8501a, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.751646 26075 consensus_queue.cc:260] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315d6e5186064f7290334850c1bcb97a" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46257 } }
I20240501 08:10:49.751698 26073 consensus_queue.cc:260] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd2ae440014564886a29d2e1d8501a" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 39405 } }
I20240501 08:10:49.751869 26075 raft_consensus.cc:388] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:49.751896 26073 raft_consensus.cc:388] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:49.751971 26075 raft_consensus.cc:482] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:49.752008 26073 raft_consensus.cc:482] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:49.752084 26075 raft_consensus.cc:3037] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.752135 26073 raft_consensus.cc:3037] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.753612 26074 raft_consensus.cc:504] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61317ad828ea4ca68919d61a4dcaa008" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 42773 } }
I20240501 08:10:49.753885 26074 leader_election.cc:302] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61317ad828ea4ca68919d61a4dcaa008; no voters:
I20240501 08:10:49.754421 26074 leader_election.cc:288] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:49.754420 26073 raft_consensus.cc:504] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd2ae440014564886a29d2e1d8501a" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 39405 } }
I20240501 08:10:49.754590 26079 raft_consensus.cc:2781] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.754853 26073 leader_election.cc:302] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cd2ae440014564886a29d2e1d8501a; no voters:
I20240501 08:10:49.755126 26075 raft_consensus.cc:504] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315d6e5186064f7290334850c1bcb97a" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46257 } }
I20240501 08:10:49.755501 26075 leader_election.cc:302] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 315d6e5186064f7290334850c1bcb97a; no voters:
I20240501 08:10:49.757820 26082 raft_consensus.cc:2781] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.758044 26073 leader_election.cc:288] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:49.764567 26083 raft_consensus.cc:2781] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.764880 26075 leader_election.cc:288] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:49.765040 26082 raft_consensus.cc:686] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 1 LEADER]: Becoming Leader. State: Replica: 56cd2ae440014564886a29d2e1d8501a, State: Running, Role: LEADER
I20240501 08:10:49.766172 26082 consensus_queue.cc:237] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd2ae440014564886a29d2e1d8501a" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 39405 } }
I20240501 08:10:49.767572 26074 ts_tablet_manager.cc:1398] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20240501 08:10:49.767762 26079 raft_consensus.cc:686] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 1 LEADER]: Becoming Leader. State: Replica: 61317ad828ea4ca68919d61a4dcaa008, State: Running, Role: LEADER
I20240501 08:10:49.768373 26073 ts_tablet_manager.cc:1398] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: Time spent starting tablet: real 0.018s user 0.006s sys 0.004s
I20240501 08:10:49.769456 26079 consensus_queue.cc:237] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61317ad828ea4ca68919d61a4dcaa008" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 42773 } }
I20240501 08:10:49.769609 26083 raft_consensus.cc:686] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 1 LEADER]: Becoming Leader. State: Replica: 315d6e5186064f7290334850c1bcb97a, State: Running, Role: LEADER
I20240501 08:10:49.770126 26083 consensus_queue.cc:237] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315d6e5186064f7290334850c1bcb97a" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46257 } }
I20240501 08:10:49.771426 26075 ts_tablet_manager.cc:1398] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: Time spent starting tablet: real 0.022s user 0.010s sys 0.008s
I20240501 08:10:49.775341 25655 catalog_manager.cc:5484] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a reported cstate change: term changed from 0 to 1, leader changed from <none> to 56cd2ae440014564886a29d2e1d8501a (127.23.244.1). New cstate: current_term: 1 leader_uuid: "56cd2ae440014564886a29d2e1d8501a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cd2ae440014564886a29d2e1d8501a" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 39405 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:49.776099 25656 catalog_manager.cc:5484] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61317ad828ea4ca68919d61a4dcaa008 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "61317ad828ea4ca68919d61a4dcaa008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61317ad828ea4ca68919d61a4dcaa008" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 42773 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:49.781996 25654 catalog_manager.cc:5484] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315d6e5186064f7290334850c1bcb97a (127.23.244.2). New cstate: current_term: 1 leader_uuid: "315d6e5186064f7290334850c1bcb97a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315d6e5186064f7290334850c1bcb97a" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46257 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:49.816839 24528 master.cc:537] Master@127.23.244.61:41719 shutting down...
I20240501 08:10:49.823035 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:49.823388 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:49.823524 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 67a64a700f5c419a889985c431c7cb7b: stopping tablet replica
I20240501 08:10:49.838389 24528 master.cc:552] Master@127.23.244.61:41719 shutdown complete.
W20240501 08:10:50.799335 26040 heartbeater.cc:644] Failed to heartbeat to 127.23.244.61:41719 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:51.310497 26100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67a64a700f5c419a889985c431c7cb7b)
I20240501 08:10:51.310701 26100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:51.311518 26101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67a64a700f5c419a889985c431c7cb7b)
I20240501 08:10:51.311625 26100 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719)
I20240501 08:10:51.311751 26101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:51.312189 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282" is_pre_election: true
I20240501 08:10:51.312414 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e6b24f64f234cdd9c86f7e2d3d20629 in term 1.
I20240501 08:10:51.312819 26101 leader_election.cc:288] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719), 8e6b24f64f234cdd9c86f7e2d3d20629 (127.23.244.60:33137)
I20240501 08:10:51.313095 25731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82143f30768b4be3959cc080edd7d282" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" is_pre_election: true
I20240501 08:10:51.313309 25731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82143f30768b4be3959cc080edd7d282 in term 1.
I20240501 08:10:51.313556 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82143f30768b4be3959cc080edd7d282, 8e6b24f64f234cdd9c86f7e2d3d20629; no voters:
I20240501 08:10:51.313722 25577 leader_election.cc:302] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82143f30768b4be3959cc080edd7d282, 8e6b24f64f234cdd9c86f7e2d3d20629; no voters:
I20240501 08:10:51.313830 26100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:10:51.313939 26100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67a64a700f5c419a889985c431c7cb7b)
I20240501 08:10:51.314015 26101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:10:51.314056 26100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:10:51.314352 26101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67a64a700f5c419a889985c431c7cb7b)
W20240501 08:10:51.314364 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:51.314503 26101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:10:51.315196 25575 leader_election.cc:334] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:51.385332 26101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:51.385396 26100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:51.386229 26101 leader_election.cc:288] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 election: Requested vote from peers 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719), 8e6b24f64f234cdd9c86f7e2d3d20629 (127.23.244.60:33137)
I20240501 08:10:51.386297 26100 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 election: Requested vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719)
I20240501 08:10:51.387006 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282"
I20240501 08:10:51.387380 25601 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e6b24f64f234cdd9c86f7e2d3d20629 in current term 2: Already voted for candidate 82143f30768b4be3959cc080edd7d282 in this term.
I20240501 08:10:51.387408 25731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82143f30768b4be3959cc080edd7d282" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629"
I20240501 08:10:51.387672 25731 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 82143f30768b4be3959cc080edd7d282 in current term 2: Already voted for candidate 8e6b24f64f234cdd9c86f7e2d3d20629 in this term.
W20240501 08:10:51.388868 25575 leader_election.cc:334] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:51.389110 25577 leader_election.cc:302] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82143f30768b4be3959cc080edd7d282; no voters: 67a64a700f5c419a889985c431c7cb7b, 8e6b24f64f234cdd9c86f7e2d3d20629
I20240501 08:10:51.389487 26101 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 08:10:51.389883 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:51.390005 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e6b24f64f234cdd9c86f7e2d3d20629; no voters: 67a64a700f5c419a889985c431c7cb7b, 82143f30768b4be3959cc080edd7d282
I20240501 08:10:51.390271 26100 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 08:10:53.160413 26112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:53.160640 26112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:53.161397 26112 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719)
I20240501 08:10:53.161870 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282" is_pre_election: true
I20240501 08:10:53.162073 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e6b24f64f234cdd9c86f7e2d3d20629 in term 2.
I20240501 08:10:53.162537 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82143f30768b4be3959cc080edd7d282, 8e6b24f64f234cdd9c86f7e2d3d20629; no voters:
W20240501 08:10:53.162634 25704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:10:53.162773 26112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:10:53.162875 26112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:53.162976 26112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:10:53.163857 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:53.227602 26112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:53.228442 26112 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 election: Requested vote from peers 82143f30768b4be3959cc080edd7d282 (127.23.244.62:45949), 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719)
I20240501 08:10:53.228986 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "82143f30768b4be3959cc080edd7d282"
I20240501 08:10:53.229228 25601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:10:53.230444 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 67a64a700f5c419a889985c431c7cb7b (127.23.244.61:41719): Network error: Client connection negotiation failed: client connection to 127.23.244.61:41719: connect: Connection refused (error 111)
I20240501 08:10:53.329607 25601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e6b24f64f234cdd9c86f7e2d3d20629 in term 3.
I20240501 08:10:53.330334 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82143f30768b4be3959cc080edd7d282, 8e6b24f64f234cdd9c86f7e2d3d20629; no voters: 67a64a700f5c419a889985c431c7cb7b
I20240501 08:10:53.330801 26112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:10:53.331220 26112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 LEADER]: Becoming Leader. State: Replica: 8e6b24f64f234cdd9c86f7e2d3d20629, State: Running, Role: LEADER
I20240501 08:10:53.331497 26112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } }
I20240501 08:10:53.333055 26115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e6b24f64f234cdd9c86f7e2d3d20629. Latest consensus state: current_term: 3 leader_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82143f30768b4be3959cc080edd7d282" member_type: VOTER last_known_addr { host: "127.23.244.62" port: 45949 } } peers { permanent_uuid: "67a64a700f5c419a889985c431c7cb7b" member_type: VOTER last_known_addr { host: "127.23.244.61" port: 41719 } } peers { permanent_uuid: "8e6b24f64f234cdd9c86f7e2d3d20629" member_type: VOTER last_known_addr { host: "127.23.244.60" port: 33137 } } }
I20240501 08:10:53.333361 26115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:53.333999 26117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:53.335837 26117 catalog_manager.cc:669] Loaded metadata for table test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0]
I20240501 08:10:53.337240 26117 tablet_loader.cc:96] loaded metadata for tablet 376f575201ce4b2aa874d89ee2153396 (table test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0])
I20240501 08:10:53.337572 26117 tablet_loader.cc:96] loaded metadata for tablet 65fc47acc3434b4185a17f49b16bd3ac (table test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0])
I20240501 08:10:53.337888 26117 tablet_loader.cc:96] loaded metadata for tablet 9d065795ef514a36948b4b7ebcab3f3c (table test-workload [id=e92dfa112aad479b89bf13ba3d8ccff0])
I20240501 08:10:53.338131 26117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:53.338940 26117 catalog_manager.cc:1259] Loaded cluster ID: cfe10519ca1948e3b7c06a382e311a95
I20240501 08:10:53.339031 26117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:53.340183 26117 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:53.341017 26117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: Loaded TSK: 0
I20240501 08:10:53.341648 26117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:53.388455 25784 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:53.388602 25784 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:53.431030 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:10:53.440554 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:53.440892 24528 tablet_replica.cc:330] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a: stopping tablet replica
I20240501 08:10:53.441079 24528 raft_consensus.cc:2229] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:53.441269 24528 raft_consensus.cc:2258] T 9d065795ef514a36948b4b7ebcab3f3c P 56cd2ae440014564886a29d2e1d8501a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:53.454811 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:10:53.457696 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:10:53.466148 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:53.466377 24528 tablet_replica.cc:330] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a: stopping tablet replica
I20240501 08:10:53.466557 24528 raft_consensus.cc:2229] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:53.466735 24528 raft_consensus.cc:2258] T 376f575201ce4b2aa874d89ee2153396 P 315d6e5186064f7290334850c1bcb97a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:53.480430 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:10:53.483070 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:10:53.491443 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:10:53.491679 24528 tablet_replica.cc:330] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008: stopping tablet replica
I20240501 08:10:53.491847 24528 raft_consensus.cc:2229] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:10:53.492164 24528 raft_consensus.cc:2258] T 65fc47acc3434b4185a17f49b16bd3ac P 61317ad828ea4ca68919d61a4dcaa008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:53.505765 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:10:53.508394 24528 master.cc:537] Master@127.23.244.62:45949 shutting down...
I20240501 08:10:53.515621 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:10:53.515825 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:53.515970 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 82143f30768b4be3959cc080edd7d282: stopping tablet replica
I20240501 08:10:53.529346 24528 master.cc:552] Master@127.23.244.62:45949 shutdown complete.
I20240501 08:10:53.533216 24528 master.cc:537] Master@127.23.244.60:33137 shutting down...
I20240501 08:10:53.541153 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 LEADER]: Raft consensus shutting down.
I20240501 08:10:53.541455 24528 pending_rounds.cc:62] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: Trying to abort 1 pending ops.
I20240501 08:10:53.541584 24528 pending_rounds.cc:68] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: Aborting op as it isn't in flight: id { term: 3 index: 11 } timestamp: 7022801114448531456 op_type: NO_OP noop_request { }
I20240501 08:10:53.541754 24528 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:10:53.541878 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:10:53.542069 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e6b24f64f234cdd9c86f7e2d3d20629: stopping tablet replica
I20240501 08:10:53.555884 24528 master.cc:552] Master@127.23.244.60:33137 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6402 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 08:10:53.568147 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:42807
I20240501 08:10:53.568639 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.571388 26118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.572402 26119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.573562 26121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.574205 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:53.574505 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:53.574579 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:53.574635 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551053574619 us; error 0 us; skew 500 ppm
I20240501 08:10:53.574815 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.576097 24528 webserver.cc:460] Webserver started at http://127.23.244.62:34411/ using document root <none> and password file <none>
I20240501 08:10:53.576350 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.576421 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.576575 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.577241 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "ad98ebf8620441e0b46985254a8ca36e"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.579577 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:53.581297 26126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.581688 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:10:53.581823 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "ad98ebf8620441e0b46985254a8ca36e"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.581938 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.589797 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.590338 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.603380 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:42807
I20240501 08:10:53.603411 26177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:42807 every 8 connection(s)
I20240501 08:10:53.604614 26178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:53.608637 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e: Bootstrap starting.
I20240501 08:10:53.609941 26178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:53.611711 26178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e: No bootstrap required, opened a new log
I20240501 08:10:53.612341 26178 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER }
I20240501 08:10:53.612479 26178 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:53.612551 26178 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad98ebf8620441e0b46985254a8ca36e, State: Initialized, Role: FOLLOWER
I20240501 08:10:53.612789 26178 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER }
I20240501 08:10:53.612962 26178 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:53.613037 26178 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:53.613140 26178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.631841 26178 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER }
I20240501 08:10:53.632167 26178 leader_election.cc:302] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad98ebf8620441e0b46985254a8ca36e; no voters:
I20240501 08:10:53.632625 26178 leader_election.cc:288] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:53.632838 26181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:53.633469 26181 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 1 LEADER]: Becoming Leader. State: Replica: ad98ebf8620441e0b46985254a8ca36e, State: Running, Role: LEADER
I20240501 08:10:53.633695 26178 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:53.633757 26181 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER }
I20240501 08:10:53.635609 26182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad98ebf8620441e0b46985254a8ca36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER } }
I20240501 08:10:53.635828 26182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:53.636229 26183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad98ebf8620441e0b46985254a8ca36e. Latest consensus state: current_term: 1 leader_uuid: "ad98ebf8620441e0b46985254a8ca36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad98ebf8620441e0b46985254a8ca36e" member_type: VOTER } }
I20240501 08:10:53.636408 26183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:53.637512 26187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:53.639919 26187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:53.640355 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:10:53.643074 26187 catalog_manager.cc:1347] Generated new cluster ID: 35e80e16d1f7438d8e5230d49491e6cd
I20240501 08:10:53.643175 26187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:53.661756 26187 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:53.662534 26187 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:53.675531 26187 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e: Generated new TSK 0
I20240501 08:10:53.675758 26187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:53.705485 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.708571 26200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.709422 26201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.710234 26203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.710433 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:53.710669 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:53.710729 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:53.710796 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551053710766 us; error 0 us; skew 500 ppm
I20240501 08:10:53.710973 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.712289 24528 webserver.cc:460] Webserver started at http://127.23.244.1:40425/ using document root <none> and password file <none>
I20240501 08:10:53.712507 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.712599 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.712756 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.713420 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.715898 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:10:53.717525 26208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.718483 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:53.718582 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.718716 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.766875 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.767513 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.768123 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.769065 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.769138 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.769263 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.769318 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.783066 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:45239
I20240501 08:10:53.783097 26270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:45239 every 8 connection(s)
I20240501 08:10:53.784956 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.788282 26275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.790032 26276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.790316 26278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.791239 24528 server_base.cc:1034] running on GCE node
I20240501 08:10:53.791532 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:53.791625 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:53.791697 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551053791674 us; error 0 us; skew 500 ppm
I20240501 08:10:53.791874 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.792943 26271 heartbeater.cc:347] Connected to a master server at 127.23.244.62:42807
I20240501 08:10:53.793067 26271 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.793368 24528 webserver.cc:460] Webserver started at http://127.23.244.2:44749/ using document root <none> and password file <none>
I20240501 08:10:53.793486 26271 heartbeater.cc:511] Master 127.23.244.62:42807 requested a full tablet report, sending...
I20240501 08:10:53.793624 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.793716 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.793876 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.794497 26143 ts_manager.cc:194] Registered new tserver with Master: f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1:45239)
I20240501 08:10:53.794734 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "6617f96b46634571a4cabd9fd303fdca"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.796000 26143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47954
I20240501 08:10:53.797685 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:53.799124 26283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.799538 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:10:53.799657 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "6617f96b46634571a4cabd9fd303fdca"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.799767 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.837992 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.838577 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.839164 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.839958 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.840034 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.840121 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.840216 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.856846 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:32799
I20240501 08:10:53.856904 26345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:32799 every 8 connection(s)
I20240501 08:10:53.858434 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.862157 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.862955 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.863201 24528 server_base.cc:1034] running on GCE node
W20240501 08:10:53.863902 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.864403 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:10:53.864493 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:10:53.864591 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551053864555 us; error 0 us; skew 500 ppm
I20240501 08:10:53.864794 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.865965 26346 heartbeater.cc:347] Connected to a master server at 127.23.244.62:42807
I20240501 08:10:53.866185 26346 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.866564 26346 heartbeater.cc:511] Master 127.23.244.62:42807 requested a full tablet report, sending...
I20240501 08:10:53.867014 24528 webserver.cc:460] Webserver started at http://127.23.244.3:45969/ using document root <none> and password file <none>
I20240501 08:10:53.867282 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.867384 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.867539 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.867550 26143 ts_manager.cc:194] Registered new tserver with Master: 6617f96b46634571a4cabd9fd303fdca (127.23.244.2:32799)
I20240501 08:10:53.868237 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "a8ecae0c2fdc486aa455270b83e8b0b5"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.868316 26143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47968
I20240501 08:10:53.870718 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:10:53.872357 26357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.872784 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:10:53.872896 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "a8ecae0c2fdc486aa455270b83e8b0b5"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-fck4"
I20240501 08:10:53.873008 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.880301 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.880885 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.881382 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.882234 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.882303 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.882411 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.882467 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.896973 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:39483
I20240501 08:10:53.897014 26419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:39483 every 8 connection(s)
I20240501 08:10:53.904014 26420 heartbeater.cc:347] Connected to a master server at 127.23.244.62:42807
I20240501 08:10:53.904158 26420 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.904497 26420 heartbeater.cc:511] Master 127.23.244.62:42807 requested a full tablet report, sending...
I20240501 08:10:53.905483 26143 ts_manager.cc:194] Registered new tserver with Master: a8ecae0c2fdc486aa455270b83e8b0b5 (127.23.244.3:39483)
I20240501 08:10:53.905764 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007996233s
I20240501 08:10:53.906570 26143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47972
I20240501 08:10:54.797271 26271 heartbeater.cc:503] Master 127.23.244.62:42807 was elected leader, sending a full tablet report...
I20240501 08:10:54.869642 26346 heartbeater.cc:503] Master 127.23.244.62:42807 was elected leader, sending a full tablet report...
I20240501 08:10:54.907904 26420 heartbeater.cc:503] Master 127.23.244.62:42807 was elected leader, sending a full tablet report...
I20240501 08:10:54.933939 24528 test_util.cc:268] Using random seed: 110370732
I20240501 08:10:54.951910 26143 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47976:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:10:54.952581 26143 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:10:54.982940 26311 tablet_service.cc:1467] Processing CreateTablet for tablet bde63d1ffa614078a4bbf3f46da72e8b (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:10:54.983027 26310 tablet_service.cc:1467] Processing CreateTablet for tablet fa5f9133cc9c48b5aa5e38910b9fb240 (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:10:54.983430 26311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde63d1ffa614078a4bbf3f46da72e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.983850 26385 tablet_service.cc:1467] Processing CreateTablet for tablet bde63d1ffa614078a4bbf3f46da72e8b (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:10:54.983930 26384 tablet_service.cc:1467] Processing CreateTablet for tablet fa5f9133cc9c48b5aa5e38910b9fb240 (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:10:54.984238 26385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde63d1ffa614078a4bbf3f46da72e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.984438 26384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5f9133cc9c48b5aa5e38910b9fb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.988634 26310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5f9133cc9c48b5aa5e38910b9fb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.992940 26440 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: Bootstrap starting.
I20240501 08:10:54.995007 26441 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: Bootstrap starting.
I20240501 08:10:54.995075 26440 tablet_bootstrap.cc:654] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.995255 26236 tablet_service.cc:1467] Processing CreateTablet for tablet bde63d1ffa614078a4bbf3f46da72e8b (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 08:10:54.995697 26236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde63d1ffa614078a4bbf3f46da72e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.995787 26235 tablet_service.cc:1467] Processing CreateTablet for tablet fa5f9133cc9c48b5aa5e38910b9fb240 (DEFAULT_TABLE table=test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 08:10:54.996374 26235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5f9133cc9c48b5aa5e38910b9fb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.997754 26441 tablet_bootstrap.cc:654] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.003007 26444 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: Bootstrap starting.
I20240501 08:10:55.003223 26441 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: No bootstrap required, opened a new log
I20240501 08:10:55.003402 26441 ts_tablet_manager.cc:1367] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240501 08:10:55.003427 26440 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: No bootstrap required, opened a new log
I20240501 08:10:55.003645 26440 ts_tablet_manager.cc:1367] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 08:10:55.004207 26440 raft_consensus.cc:348] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.004362 26440 raft_consensus.cc:374] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.004468 26440 raft_consensus.cc:729] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6617f96b46634571a4cabd9fd303fdca, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.004335 26441 raft_consensus.cc:348] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.004550 26441 raft_consensus.cc:374] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.004655 26441 raft_consensus.cc:729] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ecae0c2fdc486aa455270b83e8b0b5, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.004849 26444 tablet_bootstrap.cc:654] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.004712 26440 consensus_queue.cc:260] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.004933 26441 consensus_queue.cc:260] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.005543 26440 ts_tablet_manager.cc:1398] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:55.005812 26440 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: Bootstrap starting.
I20240501 08:10:55.006762 26441 ts_tablet_manager.cc:1398] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:55.007418 26440 tablet_bootstrap.cc:654] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.007886 26441 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: Bootstrap starting.
I20240501 08:10:55.008836 26444 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: No bootstrap required, opened a new log
I20240501 08:10:55.009047 26444 ts_tablet_manager.cc:1367] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 08:10:55.009449 26441 tablet_bootstrap.cc:654] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.009975 26444 raft_consensus.cc:348] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.010258 26444 raft_consensus.cc:374] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.010346 26444 raft_consensus.cc:729] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f6aac6d3b4c1daf8112cf7e4f1e51, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.010672 26444 consensus_queue.cc:260] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.014430 26444 ts_tablet_manager.cc:1398] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:55.014950 26441 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: No bootstrap required, opened a new log
I20240501 08:10:55.015110 26441 ts_tablet_manager.cc:1367] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:10:55.015429 26444 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Bootstrap starting.
I20240501 08:10:55.015724 26440 tablet_bootstrap.cc:492] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: No bootstrap required, opened a new log
I20240501 08:10:55.015870 26440 ts_tablet_manager.cc:1367] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240501 08:10:55.015794 26441 raft_consensus.cc:348] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.016181 26441 raft_consensus.cc:374] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.016312 26441 raft_consensus.cc:729] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ecae0c2fdc486aa455270b83e8b0b5, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.016510 26440 raft_consensus.cc:348] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.017524 26440 raft_consensus.cc:374] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.017604 26440 raft_consensus.cc:729] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6617f96b46634571a4cabd9fd303fdca, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.017616 26441 consensus_queue.cc:260] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.017848 26440 consensus_queue.cc:260] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.018349 26440 ts_tablet_manager.cc:1398] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:10:55.017079 26444 tablet_bootstrap.cc:654] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.019979 26441 ts_tablet_manager.cc:1398] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:10:55.021560 26450 raft_consensus.cc:482] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:55.021726 26450 raft_consensus.cc:504] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.022969 26444 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: No bootstrap required, opened a new log
I20240501 08:10:55.023175 26444 ts_tablet_manager.cc:1367] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 08:10:55.023432 26450 leader_election.cc:288] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6617f96b46634571a4cabd9fd303fdca (127.23.244.2:32799), a8ecae0c2fdc486aa455270b83e8b0b5 (127.23.244.3:39483)
I20240501 08:10:55.024551 26444 raft_consensus.cc:348] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.024765 26444 raft_consensus.cc:374] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.024852 26444 raft_consensus.cc:729] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f6aac6d3b4c1daf8112cf7e4f1e51, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.025076 26444 consensus_queue.cc:260] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.025578 26444 ts_tablet_manager.cc:1398] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:10:55.029259 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5f9133cc9c48b5aa5e38910b9fb240" candidate_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" is_pre_election: true
I20240501 08:10:55.029520 26395 raft_consensus.cc:2454] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f6aac6d3b4c1daf8112cf7e4f1e51 in term 0.
I20240501 08:10:55.029937 26211 leader_election.cc:302] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8ecae0c2fdc486aa455270b83e8b0b5, f71f6aac6d3b4c1daf8112cf7e4f1e51; no voters:
I20240501 08:10:55.029949 26321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5f9133cc9c48b5aa5e38910b9fb240" candidate_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6617f96b46634571a4cabd9fd303fdca" is_pre_election: true
I20240501 08:10:55.030154 26321 raft_consensus.cc:2454] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f6aac6d3b4c1daf8112cf7e4f1e51 in term 0.
I20240501 08:10:55.030171 26450 raft_consensus.cc:2781] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:55.030253 26450 raft_consensus.cc:482] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:55.030321 26450 raft_consensus.cc:3037] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.034912 26445 raft_consensus.cc:482] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:55.035053 26445 raft_consensus.cc:504] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.035800 26445 leader_election.cc:288] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ecae0c2fdc486aa455270b83e8b0b5 (127.23.244.3:39483), f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1:45239)
I20240501 08:10:55.042729 26246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde63d1ffa614078a4bbf3f46da72e8b" candidate_uuid: "6617f96b46634571a4cabd9fd303fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" is_pre_election: true
I20240501 08:10:55.043001 26246 raft_consensus.cc:2454] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6617f96b46634571a4cabd9fd303fdca in term 0.
I20240501 08:10:55.043222 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde63d1ffa614078a4bbf3f46da72e8b" candidate_uuid: "6617f96b46634571a4cabd9fd303fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" is_pre_election: true
I20240501 08:10:55.043504 26287 leader_election.cc:302] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6617f96b46634571a4cabd9fd303fdca, f71f6aac6d3b4c1daf8112cf7e4f1e51; no voters:
I20240501 08:10:55.043524 26395 raft_consensus.cc:2454] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6617f96b46634571a4cabd9fd303fdca in term 0.
I20240501 08:10:55.043789 26445 raft_consensus.cc:2781] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:55.043884 26445 raft_consensus.cc:482] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:55.043946 26445 raft_consensus.cc:3037] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.064103 26450 raft_consensus.cc:504] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.064100 26445 raft_consensus.cc:504] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.065152 26445 leader_election.cc:288] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [CANDIDATE]: Term 1 election: Requested vote from peers a8ecae0c2fdc486aa455270b83e8b0b5 (127.23.244.3:39483), f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1:45239)
I20240501 08:10:55.065358 26450 leader_election.cc:288] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [CANDIDATE]: Term 1 election: Requested vote from peers 6617f96b46634571a4cabd9fd303fdca (127.23.244.2:32799), a8ecae0c2fdc486aa455270b83e8b0b5 (127.23.244.3:39483)
I20240501 08:10:55.065455 26246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde63d1ffa614078a4bbf3f46da72e8b" candidate_uuid: "6617f96b46634571a4cabd9fd303fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51"
I20240501 08:10:55.065649 26246 raft_consensus.cc:3037] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.065722 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5f9133cc9c48b5aa5e38910b9fb240" candidate_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5"
I20240501 08:10:55.065850 26395 raft_consensus.cc:3037] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.065918 26321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5f9133cc9c48b5aa5e38910b9fb240" candidate_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6617f96b46634571a4cabd9fd303fdca"
I20240501 08:10:55.066110 26321 raft_consensus.cc:3037] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.068647 26394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde63d1ffa614078a4bbf3f46da72e8b" candidate_uuid: "6617f96b46634571a4cabd9fd303fdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5"
I20240501 08:10:55.068902 26394 raft_consensus.cc:3037] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.151785 26246 raft_consensus.cc:2454] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6617f96b46634571a4cabd9fd303fdca in term 1.
I20240501 08:10:55.151784 26321 raft_consensus.cc:2454] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f6aac6d3b4c1daf8112cf7e4f1e51 in term 1.
I20240501 08:10:55.151834 26395 raft_consensus.cc:2454] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f6aac6d3b4c1daf8112cf7e4f1e51 in term 1.
I20240501 08:10:55.152499 26287 leader_election.cc:302] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6617f96b46634571a4cabd9fd303fdca, f71f6aac6d3b4c1daf8112cf7e4f1e51; no voters:
I20240501 08:10:55.152499 26212 leader_election.cc:302] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6617f96b46634571a4cabd9fd303fdca, f71f6aac6d3b4c1daf8112cf7e4f1e51; no voters:
I20240501 08:10:55.153028 26445 raft_consensus.cc:2781] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.153523 26445 raft_consensus.cc:686] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 LEADER]: Becoming Leader. State: Replica: 6617f96b46634571a4cabd9fd303fdca, State: Running, Role: LEADER
I20240501 08:10:55.153892 26450 raft_consensus.cc:2781] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.153815 26445 consensus_queue.cc:237] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } }
I20240501 08:10:55.154314 26450 raft_consensus.cc:686] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 LEADER]: Becoming Leader. State: Replica: f71f6aac6d3b4c1daf8112cf7e4f1e51, State: Running, Role: LEADER
I20240501 08:10:55.154546 26450 consensus_queue.cc:237] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } }
I20240501 08:10:55.154668 26394 raft_consensus.cc:2454] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6617f96b46634571a4cabd9fd303fdca in term 1.
I20240501 08:10:55.156939 26143 catalog_manager.cc:5484] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca reported cstate change: term changed from 0 to 1, leader changed from <none> to 6617f96b46634571a4cabd9fd303fdca (127.23.244.2). New cstate: current_term: 1 leader_uuid: "6617f96b46634571a4cabd9fd303fdca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:55.157037 26141 catalog_manager.cc:5484] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1). New cstate: current_term: 1 leader_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:55.620805 26445 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:55.624869 26445 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:55.640331 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:55.640960 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:10:55.641062 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:55.707744 26450 consensus_queue.cc:1035] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:55.712024 26450 consensus_queue.cc:1035] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:57.641420 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:57.642036 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:10:57.642138 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:10:59.642491 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:10:59.643101 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:10:59.643203 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:01.643605 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:01.644239 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:01.644357 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:03.644807 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:03.645431 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:03.645548 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:05.645937 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:05.646540 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:05.646637 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:06.214941 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:06.218155 26475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:06.218935 26476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:06.219671 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:06.219969 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:06.221000 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:06.221073 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:06.221117 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551066221112 us; error 0 us; skew 500 ppm
I20240501 08:11:06.221261 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:06.222568 24528 webserver.cc:460] Webserver started at http://127.23.244.4:39999/ using document root <none> and password file <none>
I20240501 08:11:06.222774 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:06.222862 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:06.222985 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:06.223616 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-3-root/instance:
uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284"
format_stamp: "Formatted at 2024-05-01 08:11:06 on dist-test-slave-fck4"
I20240501 08:11:06.226068 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:06.227609 26483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:06.227970 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:11:06.228066 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-3-root
uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284"
format_stamp: "Formatted at 2024-05-01 08:11:06 on dist-test-slave-fck4"
I20240501 08:11:06.228168 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:06.247440 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:06.248001 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:06.248500 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:06.249339 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:06.249404 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:06.249475 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:06.249514 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:06.263950 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.4:34631
I20240501 08:11:06.263968 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.4:34631 every 8 connection(s)
I20240501 08:11:06.270514 26546 heartbeater.cc:347] Connected to a master server at 127.23.244.62:42807
I20240501 08:11:06.270637 26546 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:06.270937 26546 heartbeater.cc:511] Master 127.23.244.62:42807 requested a full tablet report, sending...
I20240501 08:11:06.271669 26141 ts_manager.cc:194] Registered new tserver with Master: 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4:34631)
I20240501 08:11:06.272382 26141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51416
I20240501 08:11:06.662034 26321 consensus_queue.cc:237] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } }
I20240501 08:11:06.665156 26246 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:06.665163 26394 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:06.666021 26474 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:11:06.666333 26553 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:11:06.668751 26474 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } } }
I20240501 08:11:06.671710 26394 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } } }
I20240501 08:11:06.672804 26141 catalog_manager.cc:5484] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca reported cstate change: config changed from index -1 to 2, NON_VOTER 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "6617f96b46634571a4cabd9fd303fdca" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:06.676543 26286 consensus_peers.cc:485] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca -> Peer 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4:34631): Couldn't send request to peer 4eb68a40e51a4cdfa5e96fd1e0ad2284. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde63d1ffa614078a4bbf3f46da72e8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:06.677232 26246 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } } }
I20240501 08:11:07.157344 26563 ts_tablet_manager.cc:902] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Initiating tablet copy from peer 6617f96b46634571a4cabd9fd303fdca (127.23.244.2:32799)
I20240501 08:11:07.159792 26563 tablet_copy_client.cc:308] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.2:32799
I20240501 08:11:07.166932 26331 tablet_copy_service.cc:140] P 6617f96b46634571a4cabd9fd303fdca: Received BeginTabletCopySession request for tablet bde63d1ffa614078a4bbf3f46da72e8b from peer 4eb68a40e51a4cdfa5e96fd1e0ad2284 ({username='slave'} at 127.0.0.1:53010)
I20240501 08:11:07.167173 26331 tablet_copy_service.cc:161] P 6617f96b46634571a4cabd9fd303fdca: Beginning new tablet copy session on tablet bde63d1ffa614078a4bbf3f46da72e8b from peer 4eb68a40e51a4cdfa5e96fd1e0ad2284 at {username='slave'} at 127.0.0.1:53010: session id = 4eb68a40e51a4cdfa5e96fd1e0ad2284-bde63d1ffa614078a4bbf3f46da72e8b
I20240501 08:11:07.170218 26331 tablet_copy_source_session.cc:205] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:07.171973 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde63d1ffa614078a4bbf3f46da72e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.177919 26563 tablet_copy_client.cc:783] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:07.178390 26563 tablet_copy_client.cc:647] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:07.180483 26563 tablet_copy_client.cc:523] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:07.184128 26563 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Bootstrap starting.
I20240501 08:11:07.191046 26563 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:07.191372 26563 tablet_bootstrap.cc:492] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Bootstrap complete.
I20240501 08:11:07.191563 26563 ts_tablet_manager.cc:1367] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:07.192190 26563 raft_consensus.cc:348] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } }
I20240501 08:11:07.192430 26563 raft_consensus.cc:729] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4eb68a40e51a4cdfa5e96fd1e0ad2284, State: Initialized, Role: LEARNER
I20240501 08:11:07.192610 26563 consensus_queue.cc:260] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: true } }
I20240501 08:11:07.193320 26546 heartbeater.cc:503] Master 127.23.244.62:42807 was elected leader, sending a full tablet report...
I20240501 08:11:07.193364 26563 ts_tablet_manager.cc:1398] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:07.194027 26331 tablet_copy_service.cc:342] P 6617f96b46634571a4cabd9fd303fdca: Request end of tablet copy session 4eb68a40e51a4cdfa5e96fd1e0ad2284-bde63d1ffa614078a4bbf3f46da72e8b received from {username='slave'} at 127.0.0.1:53010
I20240501 08:11:07.194152 26331 tablet_copy_service.cc:433] P 6617f96b46634571a4cabd9fd303fdca: ending tablet copy session 4eb68a40e51a4cdfa5e96fd1e0ad2284-bde63d1ffa614078a4bbf3f46da72e8b on tablet bde63d1ffa614078a4bbf3f46da72e8b with peer 4eb68a40e51a4cdfa5e96fd1e0ad2284
I20240501 08:11:07.534724 26521 raft_consensus.cc:1203] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:11:07.646936 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:07.647496 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:07.647626 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:08.033340 26571 raft_consensus.cc:1050] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: attempting to promote NON_VOTER 4eb68a40e51a4cdfa5e96fd1e0ad2284 to VOTER
I20240501 08:11:08.034235 26571 consensus_queue.cc:237] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } }
I20240501 08:11:08.038007 26521 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 LEARNER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:08.038026 26246 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:08.038763 26572 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:08.039299 26572 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:08.047690 26394 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:08.048362 26571 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:08.048184 26572 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.048939 26246 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.053336 26521 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.052505 26394 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.054474 26141 catalog_manager.cc:5484] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca reported cstate change: config changed from index 2 to 3, 4eb68a40e51a4cdfa5e96fd1e0ad2284 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6617f96b46634571a4cabd9fd303fdca" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f71f6aac6d3b4c1daf8112cf7e4f1e51" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 45239 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:08.450848 26321 consensus_queue.cc:237] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } }
I20240501 08:11:08.452430 26521 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:08.452431 26394 raft_consensus.cc:1261] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Refusing update from remote peer 6617f96b46634571a4cabd9fd303fdca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:08.452929 26574 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:08.453217 26572 consensus_queue.cc:1035] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:08.455415 26572 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.455847 26394 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.455880 26521 raft_consensus.cc:2932] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.459223 26130 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde63d1ffa614078a4bbf3f46da72e8b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:08.460317 26142 catalog_manager.cc:5484] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 reported cstate change: config changed from index 3 to 4, VOTER f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1) evicted. New cstate: current_term: 1 leader_uuid: "6617f96b46634571a4cabd9fd303fdca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6617f96b46634571a4cabd9fd303fdca" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 32799 } } peers { permanent_uuid: "a8ecae0c2fdc486aa455270b83e8b0b5" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39483 } } peers { permanent_uuid: "4eb68a40e51a4cdfa5e96fd1e0ad2284" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34631 } attrs { promote: false } } }
I20240501 08:11:08.463955 26235 tablet_service.cc:1514] Processing DeleteTablet for tablet bde63d1ffa614078a4bbf3f46da72e8b with delete_type TABLET_DATA_TOMBSTONED (TS f71f6aac6d3b4c1daf8112cf7e4f1e51 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36934
I20240501 08:11:08.464835 26584 tablet_replica.cc:330] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: stopping tablet replica
I20240501 08:11:08.465011 26584 raft_consensus.cc:2229] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:08.465235 26584 raft_consensus.cc:2258] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:08.465922 26584 ts_tablet_manager.cc:1875] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:08.469959 26584 ts_tablet_manager.cc:1888] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:11:08.470098 26584 log.cc:1196] T bde63d1ffa614078a4bbf3f46da72e8b P f71f6aac6d3b4c1daf8112cf7e4f1e51: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/bde63d1ffa614078a4bbf3f46da72e8b
I20240501 08:11:08.470708 26130 catalog_manager.cc:4830] TS f71f6aac6d3b4c1daf8112cf7e4f1e51 (127.23.244.1:45239): tablet bde63d1ffa614078a4bbf3f46da72e8b (table test-workload [id=9cb259175d2b4f508700d8c14bd1afb2]) successfully deleted
I20240501 08:11:09.648078 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:09.648696 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:09.648846 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:11.649242 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:11.649819 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:11.649924 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:13.650350 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:13.651019 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:13.651134 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:15.651583 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:15.652298 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:15.652419 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:17.652812 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:17.653414 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:17.653529 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:19.653950 26197 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:19.654572 26197 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:19.654690 26197 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:20.346765 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:11:20.360219 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:20.360718 24528 tablet_replica.cc:330] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51: stopping tablet replica
I20240501 08:11:20.360978 24528 raft_consensus.cc:2229] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:20.361344 24528 raft_consensus.cc:2258] T fa5f9133cc9c48b5aa5e38910b9fb240 P f71f6aac6d3b4c1daf8112cf7e4f1e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.377038 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:11:20.380052 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:11:20.389174 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:20.389410 24528 tablet_replica.cc:330] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca: stopping tablet replica
I20240501 08:11:20.389597 24528 raft_consensus.cc:2229] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:20.389766 24528 raft_consensus.cc:2258] T fa5f9133cc9c48b5aa5e38910b9fb240 P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.390301 24528 tablet_replica.cc:330] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca: stopping tablet replica
I20240501 08:11:20.390412 24528 raft_consensus.cc:2229] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:20.390663 24528 raft_consensus.cc:2258] T bde63d1ffa614078a4bbf3f46da72e8b P 6617f96b46634571a4cabd9fd303fdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.406847 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:11:20.410147 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:11:20.419291 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:20.419541 24528 tablet_replica.cc:330] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5: stopping tablet replica
I20240501 08:11:20.419751 24528 raft_consensus.cc:2229] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:20.420024 24528 raft_consensus.cc:2258] T bde63d1ffa614078a4bbf3f46da72e8b P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.420663 24528 tablet_replica.cc:330] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5: stopping tablet replica
I20240501 08:11:20.420871 24528 raft_consensus.cc:2229] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:20.421051 24528 raft_consensus.cc:2258] T fa5f9133cc9c48b5aa5e38910b9fb240 P a8ecae0c2fdc486aa455270b83e8b0b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.434839 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:11:20.437803 24528 tablet_server.cc:178] TabletServer@127.23.244.4:0 shutting down...
I20240501 08:11:20.446309 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:20.446633 24528 tablet_replica.cc:330] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284: stopping tablet replica
I20240501 08:11:20.446861 24528 raft_consensus.cc:2229] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:20.447105 24528 raft_consensus.cc:2258] T bde63d1ffa614078a4bbf3f46da72e8b P 4eb68a40e51a4cdfa5e96fd1e0ad2284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.466150 24528 tablet_server.cc:194] TabletServer@127.23.244.4:0 shutdown complete.
I20240501 08:11:20.469003 24528 master.cc:537] Master@127.23.244.62:42807 shutting down...
I20240501 08:11:20.476974 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:20.477173 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.477260 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P ad98ebf8620441e0b46985254a8ca36e: stopping tablet replica
I20240501 08:11:20.490248 24528 master.cc:552] Master@127.23.244.62:42807 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26933 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 08:11:20.501588 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:37439
I20240501 08:11:20.502056 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:20.504884 26587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:20.505443 26588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:20.506171 26590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:20.506927 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:20.507122 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:20.507174 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:20.507205 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551080507201 us; error 0 us; skew 500 ppm
I20240501 08:11:20.507349 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:20.513713 24528 webserver.cc:460] Webserver started at http://127.23.244.62:43199/ using document root <none> and password file <none>
I20240501 08:11:20.513907 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:20.513958 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:20.514060 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:20.514657 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "922a0a567a2346eb8d62f4af7489f7e4"
format_stamp: "Formatted at 2024-05-01 08:11:20 on dist-test-slave-fck4"
I20240501 08:11:20.516986 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:11:20.518379 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:20.518728 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:11:20.518815 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "922a0a567a2346eb8d62f4af7489f7e4"
format_stamp: "Formatted at 2024-05-01 08:11:20 on dist-test-slave-fck4"
I20240501 08:11:20.518929 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:20.553285 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:20.553761 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:20.565513 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:37439
I20240501 08:11:20.565543 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:37439 every 8 connection(s)
I20240501 08:11:20.566803 26647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:20.570976 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4: Bootstrap starting.
I20240501 08:11:20.572077 26647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:20.573565 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4: No bootstrap required, opened a new log
I20240501 08:11:20.574193 26647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER }
I20240501 08:11:20.574326 26647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:20.574400 26647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922a0a567a2346eb8d62f4af7489f7e4, State: Initialized, Role: FOLLOWER
I20240501 08:11:20.574643 26647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER }
I20240501 08:11:20.574786 26647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:20.574858 26647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:20.574951 26647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:20.576656 26647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER }
I20240501 08:11:20.576871 26647 leader_election.cc:302] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 922a0a567a2346eb8d62f4af7489f7e4; no voters:
I20240501 08:11:20.577245 26647 leader_election.cc:288] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:20.577358 26650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:20.577795 26650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 1 LEADER]: Becoming Leader. State: Replica: 922a0a567a2346eb8d62f4af7489f7e4, State: Running, Role: LEADER
I20240501 08:11:20.578068 26650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER }
I20240501 08:11:20.578110 26647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:20.578773 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 922a0a567a2346eb8d62f4af7489f7e4. Latest consensus state: current_term: 1 leader_uuid: "922a0a567a2346eb8d62f4af7489f7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER } }
I20240501 08:11:20.578997 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:20.579111 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "922a0a567a2346eb8d62f4af7489f7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a0a567a2346eb8d62f4af7489f7e4" member_type: VOTER } }
I20240501 08:11:20.579263 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:20.580047 26655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:20.582006 26655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:20.583016 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:11:20.585142 26655 catalog_manager.cc:1347] Generated new cluster ID: 76d6e386f6d44b77b26bd5195fbd2a38
I20240501 08:11:20.585223 26655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:20.591810 26655 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:20.592588 26655 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:20.600270 26655 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4: Generated new TSK 0
I20240501 08:11:20.600486 26655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:20.615427 24528 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 9.6621e-05s
I20240501 08:11:32.697122 24528 master.cc:537] Master@127.23.244.62:37439 shutting down...
I20240501 08:11:32.708972 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:32.709200 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:32.709276 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 922a0a567a2346eb8d62f4af7489f7e4: stopping tablet replica
I20240501 08:11:32.722833 24528 master.cc:552] Master@127.23.244.62:37439 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12228 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 08:11:32.730496 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:34225
I20240501 08:11:32.730960 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:32.734084 26669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.734727 26670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.735258 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:32.736059 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:32.737960 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:32.738026 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:32.738059 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551092738046 us; error 0 us; skew 500 ppm
I20240501 08:11:32.738188 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:32.745635 24528 webserver.cc:460] Webserver started at http://127.23.244.62:42529/ using document root <none> and password file <none>
I20240501 08:11:32.745846 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:32.745913 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:32.746016 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:32.746601 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "88ebf2314c95421693fd6004b4cd3a0f"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.749063 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:32.750480 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:32.750854 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:11:32.750970 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "88ebf2314c95421693fd6004b4cd3a0f"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.751080 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:32.801331 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:32.801867 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:32.811861 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:34225
I20240501 08:11:32.811872 26728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:34225 every 8 connection(s)
I20240501 08:11:32.813110 26729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:32.816915 26729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: Bootstrap starting.
I20240501 08:11:32.818002 26729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:32.819425 26729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: No bootstrap required, opened a new log
I20240501 08:11:32.819990 26729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER }
I20240501 08:11:32.820115 26729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:32.820168 26729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ebf2314c95421693fd6004b4cd3a0f, State: Initialized, Role: FOLLOWER
I20240501 08:11:32.820345 26729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER }
I20240501 08:11:32.820446 26729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:32.820497 26729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:32.820562 26729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:32.822249 26729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER }
I20240501 08:11:32.822418 26729 leader_election.cc:302] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ebf2314c95421693fd6004b4cd3a0f; no voters:
I20240501 08:11:32.822751 26729 leader_election.cc:288] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:32.822865 26732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:32.823341 26732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 1 LEADER]: Becoming Leader. State: Replica: 88ebf2314c95421693fd6004b4cd3a0f, State: Running, Role: LEADER
I20240501 08:11:32.823623 26729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:32.823578 26732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER }
I20240501 08:11:32.824415 26733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ebf2314c95421693fd6004b4cd3a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER } }
I20240501 08:11:32.824518 26734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ebf2314c95421693fd6004b4cd3a0f. Latest consensus state: current_term: 1 leader_uuid: "88ebf2314c95421693fd6004b4cd3a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ebf2314c95421693fd6004b4cd3a0f" member_type: VOTER } }
I20240501 08:11:32.824748 26733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:32.824815 26734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:32.832033 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20240501 08:11:32.832875 26748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:11:32.832957 26748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:11:32.833065 26738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:32.834121 26738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:32.836794 26738 catalog_manager.cc:1347] Generated new cluster ID: d2d673f1fed04735a504518b31a5f27d
I20240501 08:11:32.836863 26738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:32.856159 26738 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:32.856982 26738 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:32.868446 26738 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: Generated new TSK 0
I20240501 08:11:32.868688 26738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:32.897236 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:32.900435 26750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.901116 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:32.901783 24528 server_base.cc:1034] running on GCE node
W20240501 08:11:32.902029 26753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:32.902364 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:32.902433 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:32.902475 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551092902462 us; error 0 us; skew 500 ppm
I20240501 08:11:32.902595 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:32.903834 24528 webserver.cc:460] Webserver started at http://127.23.244.1:35649/ using document root <none> and password file <none>
I20240501 08:11:32.904045 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:32.904122 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:32.904229 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:32.904881 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "b9722c49fdd84e38b8545982f23064a6"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.907266 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:32.908756 26758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:32.909133 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:11:32.909229 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "b9722c49fdd84e38b8545982f23064a6"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.909320 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:32.948050 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:32.948637 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:32.949213 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:32.957746 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:32.957813 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:32.957870 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:32.957921 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:32.969746 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:45715
I20240501 08:11:32.969764 26820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:45715 every 8 connection(s)
I20240501 08:11:32.971436 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:32.974644 26825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.975292 26826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.975538 26829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:32.976150 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:32.976394 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:32.976451 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:32.976485 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551092976481 us; error 0 us; skew 500 ppm
I20240501 08:11:32.976616 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:32.978127 24528 webserver.cc:460] Webserver started at http://127.23.244.2:41305/ using document root <none> and password file <none>
I20240501 08:11:32.978340 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:32.978415 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:32.978533 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:32.978601 26821 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34225
I20240501 08:11:32.978718 26821 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:32.979018 26821 heartbeater.cc:511] Master 127.23.244.62:34225 requested a full tablet report, sending...
I20240501 08:11:32.979290 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "a128699ee5ba481d946e0ad0e2a19a0c"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.979856 26694 ts_manager.cc:194] Registered new tserver with Master: b9722c49fdd84e38b8545982f23064a6 (127.23.244.1:45715)
I20240501 08:11:32.980836 26694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58310
I20240501 08:11:32.982062 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240501 08:11:32.983551 26833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:32.983868 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:11:32.983968 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "a128699ee5ba481d946e0ad0e2a19a0c"
format_stamp: "Formatted at 2024-05-01 08:11:32 on dist-test-slave-fck4"
I20240501 08:11:32.984074 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:32.999270 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:32.999724 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:33.000217 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:33.001008 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:33.001063 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:33.001139 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:33.001175 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:33.014287 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:40099
I20240501 08:11:33.014308 26895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:40099 every 8 connection(s)
I20240501 08:11:33.015877 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:33.018664 26899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.020782 26902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.022060 26900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:33.023897 26896 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34225
I20240501 08:11:33.024065 26896 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:33.024639 26896 heartbeater.cc:511] Master 127.23.244.62:34225 requested a full tablet report, sending...
I20240501 08:11:33.026221 26694 ts_manager.cc:194] Registered new tserver with Master: a128699ee5ba481d946e0ad0e2a19a0c (127.23.244.2:40099)
I20240501 08:11:33.027040 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:33.027283 26694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58312
I20240501 08:11:33.027318 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:33.027465 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:33.027518 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551093027514 us; error 0 us; skew 500 ppm
I20240501 08:11:33.027712 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:33.029106 24528 webserver.cc:460] Webserver started at http://127.23.244.3:43217/ using document root <none> and password file <none>
I20240501 08:11:33.029304 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:33.029404 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:33.029520 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:33.030165 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "a473c984725746b0be2a44e9d7e464e3"
format_stamp: "Formatted at 2024-05-01 08:11:33 on dist-test-slave-fck4"
I20240501 08:11:33.032694 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:11:33.034710 26907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:33.035375 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:11:33.035495 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "a473c984725746b0be2a44e9d7e464e3"
format_stamp: "Formatted at 2024-05-01 08:11:33 on dist-test-slave-fck4"
I20240501 08:11:33.035635 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:33.061611 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:33.062331 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:33.063051 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:33.064124 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:33.064188 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:33.064275 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:33.064317 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:33.076659 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:37989
I20240501 08:11:33.076683 26969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:37989 every 8 connection(s)
I20240501 08:11:33.083429 26970 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34225
I20240501 08:11:33.083546 26970 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:33.083911 26970 heartbeater.cc:511] Master 127.23.244.62:34225 requested a full tablet report, sending...
I20240501 08:11:33.084717 26694 ts_manager.cc:194] Registered new tserver with Master: a473c984725746b0be2a44e9d7e464e3 (127.23.244.3:37989)
I20240501 08:11:33.085440 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0078599s
I20240501 08:11:33.085536 26694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58318
I20240501 08:11:33.982169 26821 heartbeater.cc:503] Master 127.23.244.62:34225 was elected leader, sending a full tablet report...
I20240501 08:11:34.028813 26896 heartbeater.cc:503] Master 127.23.244.62:34225 was elected leader, sending a full tablet report...
I20240501 08:11:34.086882 26970 heartbeater.cc:503] Master 127.23.244.62:34225 was elected leader, sending a full tablet report...
I20240501 08:11:34.831979 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:36.832347 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:38.832768 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:40.833212 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:42.833629 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:44.834049 26747 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:45.166749 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:11:45.179908 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:45.194072 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:11:45.196678 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:11:45.204612 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:45.217264 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:11:45.220041 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:11:45.229104 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:45.242074 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:11:45.244755 24528 master.cc:537] Master@127.23.244.62:34225 shutting down...
I20240501 08:11:45.252046 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:45.252300 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:45.252410 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88ebf2314c95421693fd6004b4cd3a0f: stopping tablet replica
I20240501 08:11:45.265533 24528 master.cc:552] Master@127.23.244.62:34225 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12544 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 08:11:45.274605 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:34127
I20240501 08:11:45.275081 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.277667 26974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.278440 26975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.279136 26977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.279762 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:45.279989 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:45.280051 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:45.280107 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551105280089 us; error 0 us; skew 500 ppm
I20240501 08:11:45.280254 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.281678 24528 webserver.cc:460] Webserver started at http://127.23.244.62:45073/ using document root <none> and password file <none>
I20240501 08:11:45.281942 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.282028 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.282168 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.282881 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "9e3f230e2f5c4705b790612eb398d55f"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.285513 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 08:11:45.287148 26982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.287520 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:11:45.287640 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "9e3f230e2f5c4705b790612eb398d55f"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.287770 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.299085 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.299602 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.311385 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:34127
I20240501 08:11:45.311414 27033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:34127 every 8 connection(s)
I20240501 08:11:45.312711 27034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.316699 27034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f: Bootstrap starting.
I20240501 08:11:45.317945 27034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.319475 27034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f: No bootstrap required, opened a new log
I20240501 08:11:45.320029 27034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER }
I20240501 08:11:45.320158 27034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.320209 27034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3f230e2f5c4705b790612eb398d55f, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.320410 27034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER }
I20240501 08:11:45.320544 27034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:45.320598 27034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:45.320664 27034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.322325 27034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER }
I20240501 08:11:45.322506 27034 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3f230e2f5c4705b790612eb398d55f; no voters:
I20240501 08:11:45.322849 27034 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:45.322970 27037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:45.323424 27037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 1 LEADER]: Becoming Leader. State: Replica: 9e3f230e2f5c4705b790612eb398d55f, State: Running, Role: LEADER
I20240501 08:11:45.323634 27034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:45.323639 27037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER }
I20240501 08:11:45.324537 27038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3f230e2f5c4705b790612eb398d55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER } }
I20240501 08:11:45.324635 27039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3f230e2f5c4705b790612eb398d55f. Latest consensus state: current_term: 1 leader_uuid: "9e3f230e2f5c4705b790612eb398d55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3f230e2f5c4705b790612eb398d55f" member_type: VOTER } }
I20240501 08:11:45.324905 27039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:45.328577 27038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:45.329464 27043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:45.329964 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:11:45.330950 27043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:45.333542 27043 catalog_manager.cc:1347] Generated new cluster ID: 12d51773f75a4e69ab05ab89295cec2f
I20240501 08:11:45.333631 27043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:45.344760 27043 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:45.345547 27043 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:45.354420 27043 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f: Generated new TSK 0
I20240501 08:11:45.354619 27043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:45.362520 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.365240 27055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.366066 27056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.366319 27058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.367117 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:45.367440 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:45.367558 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:45.367624 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551105367606 us; error 0 us; skew 500 ppm
I20240501 08:11:45.367772 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.369313 24528 webserver.cc:460] Webserver started at http://127.23.244.1:44171/ using document root <none> and password file <none>
I20240501 08:11:45.369557 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.369633 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.369769 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.370417 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "9af9b41aafab4bf3922e773924293b0e"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.372798 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:45.374258 27063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.374583 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:11:45.374701 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "9af9b41aafab4bf3922e773924293b0e"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.374812 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.401749 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.402272 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.402819 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.403604 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.403662 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.403739 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.403772 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.414171 27125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:35881 every 8 connection(s)
I20240501 08:11:45.414150 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:35881
I20240501 08:11:45.415776 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.419070 27131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.419250 27130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.419999 27133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.420751 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:45.420972 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:45.421027 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:45.421070 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551105421066 us; error 0 us; skew 500 ppm
I20240501 08:11:45.421178 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.422057 27126 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34127
I20240501 08:11:45.422180 27126 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.422502 27126 heartbeater.cc:511] Master 127.23.244.62:34127 requested a full tablet report, sending...
I20240501 08:11:45.422571 24528 webserver.cc:460] Webserver started at http://127.23.244.2:42961/ using document root <none> and password file <none>
I20240501 08:11:45.422793 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.422856 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.422966 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.423343 26999 ts_manager.cc:194] Registered new tserver with Master: 9af9b41aafab4bf3922e773924293b0e (127.23.244.1:35881)
I20240501 08:11:45.423571 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "ec82d6f2124e40268d0d7413fb9569b2"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.424305 26999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44704
I20240501 08:11:45.426095 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:45.427634 27138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.427981 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:11:45.428062 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "ec82d6f2124e40268d0d7413fb9569b2"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.428145 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.451318 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.451797 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.452255 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.453038 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.453090 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.453155 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.453197 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.463953 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:40071
I20240501 08:11:45.463976 27200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:40071 every 8 connection(s)
I20240501 08:11:45.465399 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.468585 27205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.468837 27204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.469467 27207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.470016 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:45.470259 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:45.470322 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:45.470346 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551105470343 us; error 0 us; skew 500 ppm
I20240501 08:11:45.470500 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.471786 27201 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34127
I20240501 08:11:45.471928 27201 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.472256 27201 heartbeater.cc:511] Master 127.23.244.62:34127 requested a full tablet report, sending...
I20240501 08:11:45.472617 24528 webserver.cc:460] Webserver started at http://127.23.244.3:35159/ using document root <none> and password file <none>
I20240501 08:11:45.472848 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.472921 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.473031 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.473122 26999 ts_manager.cc:194] Registered new tserver with Master: ec82d6f2124e40268d0d7413fb9569b2 (127.23.244.2:40071)
I20240501 08:11:45.473767 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "57e60960cc924746b23b2df3412c8b4e"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.474067 26999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44718
I20240501 08:11:45.476199 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:45.477636 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.477924 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:11:45.477998 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "57e60960cc924746b23b2df3412c8b4e"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.478089 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.494957 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.495398 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.495872 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.496579 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.496628 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.496692 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.496763 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.506753 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:44635
I20240501 08:11:45.506769 27274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:44635 every 8 connection(s)
I20240501 08:11:45.508169 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.511314 27278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.511675 27279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.512305 27281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.512436 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:45.512878 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:45.512934 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:45.512971 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551105512967 us; error 0 us; skew 500 ppm
I20240501 08:11:45.513109 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.514436 24528 webserver.cc:460] Webserver started at http://127.23.244.4:36489/ using document root <none> and password file <none>
I20240501 08:11:45.514653 27275 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34127
I20240501 08:11:45.514678 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.514784 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.514801 27275 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.514897 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.515180 27275 heartbeater.cc:511] Master 127.23.244.62:34127 requested a full tablet report, sending...
I20240501 08:11:45.515784 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-3-root/instance:
uuid: "9784a11aa8204772b104cf73eec70812"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.516041 26999 ts_manager.cc:194] Registered new tserver with Master: 57e60960cc924746b23b2df3412c8b4e (127.23.244.3:44635)
I20240501 08:11:45.517050 26999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44720
I20240501 08:11:45.518630 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:45.519925 27286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.520193 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:11:45.520273 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-3-root
uuid: "9784a11aa8204772b104cf73eec70812"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-fck4"
I20240501 08:11:45.520391 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714551036644763-24528-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.534487 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.534914 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.535351 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.536036 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.536084 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.536156 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.536188 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.546208 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.4:34919
I20240501 08:11:45.546247 27348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.4:34919 every 8 connection(s)
I20240501 08:11:45.552317 27349 heartbeater.cc:347] Connected to a master server at 127.23.244.62:34127
I20240501 08:11:45.552433 27349 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.552788 27349 heartbeater.cc:511] Master 127.23.244.62:34127 requested a full tablet report, sending...
I20240501 08:11:45.553498 26999 ts_manager.cc:194] Registered new tserver with Master: 9784a11aa8204772b104cf73eec70812 (127.23.244.4:34919)
I20240501 08:11:45.553846 24528 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006795055s
I20240501 08:11:45.554330 26999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44732
I20240501 08:11:46.425447 27126 heartbeater.cc:503] Master 127.23.244.62:34127 was elected leader, sending a full tablet report...
I20240501 08:11:46.475225 27201 heartbeater.cc:503] Master 127.23.244.62:34127 was elected leader, sending a full tablet report...
I20240501 08:11:46.518343 27275 heartbeater.cc:503] Master 127.23.244.62:34127 was elected leader, sending a full tablet report...
I20240501 08:11:46.555568 27349 heartbeater.cc:503] Master 127.23.244.62:34127 was elected leader, sending a full tablet report...
I20240501 08:11:46.588573 24528 test_util.cc:268] Using random seed: 162025368
I20240501 08:11:46.599735 26999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44750:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 08:11:46.628957 27240 tablet_service.cc:1467] Processing CreateTablet for tablet 23fe2cfb39084c3ebd3c9d47502da2ba (DEFAULT_TABLE table=test-workload [id=9fa1bc7562f846e5a3b9bdc3598dbf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.629154 27314 tablet_service.cc:1467] Processing CreateTablet for tablet 23fe2cfb39084c3ebd3c9d47502da2ba (DEFAULT_TABLE table=test-workload [id=9fa1bc7562f846e5a3b9bdc3598dbf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.629491 27314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fe2cfb39084c3ebd3c9d47502da2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.629525 27240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fe2cfb39084c3ebd3c9d47502da2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.629441 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 23fe2cfb39084c3ebd3c9d47502da2ba (DEFAULT_TABLE table=test-workload [id=9fa1bc7562f846e5a3b9bdc3598dbf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.629825 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fe2cfb39084c3ebd3c9d47502da2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.635466 27370 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: Bootstrap starting.
I20240501 08:11:46.635928 27371 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: Bootstrap starting.
I20240501 08:11:46.636523 27372 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: Bootstrap starting.
I20240501 08:11:46.637373 27370 tablet_bootstrap.cc:654] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.637828 27372 tablet_bootstrap.cc:654] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.638207 27371 tablet_bootstrap.cc:654] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.639237 27370 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: No bootstrap required, opened a new log
I20240501 08:11:46.639379 27370 ts_tablet_manager.cc:1367] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:46.640237 27370 raft_consensus.cc:348] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.640406 27371 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: No bootstrap required, opened a new log
I20240501 08:11:46.640439 27370 raft_consensus.cc:374] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.640491 27370 raft_consensus.cc:729] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e60960cc924746b23b2df3412c8b4e, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.640518 27371 ts_tablet_manager.cc:1367] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:46.640859 27370 consensus_queue.cc:260] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.641320 27371 raft_consensus.cc:348] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.641649 27371 raft_consensus.cc:374] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.641732 27370 ts_tablet_manager.cc:1398] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:46.641757 27371 raft_consensus.cc:729] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9784a11aa8204772b104cf73eec70812, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.642021 27371 consensus_queue.cc:260] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.643056 27371 ts_tablet_manager.cc:1398] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:11:46.643128 27372 tablet_bootstrap.cc:492] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: No bootstrap required, opened a new log
I20240501 08:11:46.643292 27372 ts_tablet_manager.cc:1367] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:11:46.644567 27372 raft_consensus.cc:348] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.644863 27372 raft_consensus.cc:374] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.644945 27372 raft_consensus.cc:729] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec82d6f2124e40268d0d7413fb9569b2, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.645174 27372 consensus_queue.cc:260] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.645932 27372 ts_tablet_manager.cc:1398] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 08:11:46.716008 27202 tablet.cc:2369] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:46.722633 27378 raft_consensus.cc:482] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.722803 27378 raft_consensus.cc:504] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.723670 27378 leader_election.cc:288] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9784a11aa8204772b104cf73eec70812 (127.23.244.4:34919), 57e60960cc924746b23b2df3412c8b4e (127.23.244.3:44635)
I20240501 08:11:46.729823 27250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fe2cfb39084c3ebd3c9d47502da2ba" candidate_uuid: "ec82d6f2124e40268d0d7413fb9569b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e60960cc924746b23b2df3412c8b4e" is_pre_election: true
I20240501 08:11:46.730070 27250 raft_consensus.cc:2454] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec82d6f2124e40268d0d7413fb9569b2 in term 0.
I20240501 08:11:46.730013 27324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fe2cfb39084c3ebd3c9d47502da2ba" candidate_uuid: "ec82d6f2124e40268d0d7413fb9569b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9784a11aa8204772b104cf73eec70812" is_pre_election: true
I20240501 08:11:46.730268 27324 raft_consensus.cc:2454] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec82d6f2124e40268d0d7413fb9569b2 in term 0.
I20240501 08:11:46.730455 27139 leader_election.cc:302] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e60960cc924746b23b2df3412c8b4e, ec82d6f2124e40268d0d7413fb9569b2; no voters:
I20240501 08:11:46.730698 27378 raft_consensus.cc:2781] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.730777 27378 raft_consensus.cc:482] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.730852 27378 raft_consensus.cc:3037] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.732668 27378 raft_consensus.cc:504] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.733160 27378 leader_election.cc:288] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9784a11aa8204772b104cf73eec70812 (127.23.244.4:34919), 57e60960cc924746b23b2df3412c8b4e (127.23.244.3:44635)
I20240501 08:11:46.733587 27324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fe2cfb39084c3ebd3c9d47502da2ba" candidate_uuid: "ec82d6f2124e40268d0d7413fb9569b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9784a11aa8204772b104cf73eec70812"
I20240501 08:11:46.733611 27250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fe2cfb39084c3ebd3c9d47502da2ba" candidate_uuid: "ec82d6f2124e40268d0d7413fb9569b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e60960cc924746b23b2df3412c8b4e"
I20240501 08:11:46.733727 27324 raft_consensus.cc:3037] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.733758 27250 raft_consensus.cc:3037] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.735527 27250 raft_consensus.cc:2454] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec82d6f2124e40268d0d7413fb9569b2 in term 1.
I20240501 08:11:46.735528 27324 raft_consensus.cc:2454] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec82d6f2124e40268d0d7413fb9569b2 in term 1.
I20240501 08:11:46.735878 27142 leader_election.cc:302] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9784a11aa8204772b104cf73eec70812, ec82d6f2124e40268d0d7413fb9569b2; no voters:
I20240501 08:11:46.736107 27378 raft_consensus.cc:2781] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.736487 27378 raft_consensus.cc:686] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 1 LEADER]: Becoming Leader. State: Replica: ec82d6f2124e40268d0d7413fb9569b2, State: Running, Role: LEADER
I20240501 08:11:46.736771 27378 consensus_queue.cc:237] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } }
I20240501 08:11:46.739152 26998 catalog_manager.cc:5484] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec82d6f2124e40268d0d7413fb9569b2 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "ec82d6f2124e40268d0d7413fb9569b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec82d6f2124e40268d0d7413fb9569b2" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 40071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:47.208457 27378 consensus_queue.cc:1035] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e60960cc924746b23b2df3412c8b4e" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:47.213012 27381 consensus_queue.cc:1035] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9784a11aa8204772b104cf73eec70812" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 34919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:47.328642 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:47.329083 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:47.329208 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:49.329664 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:49.330144 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:49.330267 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:51.330691 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:51.331157 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:51.331276 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:53.331678 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:53.332142 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:53.332263 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:55.332686 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:55.333182 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:55.333300 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:57.333730 27052 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:57.334208 27052 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:57.334328 27052 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:11:57.798399 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:11:57.809031 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:57.821849 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:11:57.824247 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:11:57.831707 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:57.832003 24528 tablet_replica.cc:330] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2: stopping tablet replica
I20240501 08:11:57.832180 24528 raft_consensus.cc:2229] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:57.832482 24528 raft_consensus.cc:2258] T 23fe2cfb39084c3ebd3c9d47502da2ba P ec82d6f2124e40268d0d7413fb9569b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:57.846377 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:11:57.849225 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:11:57.857975 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:57.858229 24528 tablet_replica.cc:330] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e: stopping tablet replica
I20240501 08:11:57.858417 24528 raft_consensus.cc:2229] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:57.858625 24528 raft_consensus.cc:2258] T 23fe2cfb39084c3ebd3c9d47502da2ba P 57e60960cc924746b23b2df3412c8b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:57.880970 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:11:57.883716 24528 tablet_server.cc:178] TabletServer@127.23.244.4:0 shutting down...
I20240501 08:11:57.892189 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:11:57.892410 24528 tablet_replica.cc:330] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812: stopping tablet replica
I20240501 08:11:57.892582 24528 raft_consensus.cc:2229] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:57.892769 24528 raft_consensus.cc:2258] T 23fe2cfb39084c3ebd3c9d47502da2ba P 9784a11aa8204772b104cf73eec70812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:57.905880 24528 tablet_server.cc:194] TabletServer@127.23.244.4:0 shutdown complete.
I20240501 08:11:57.908705 24528 master.cc:537] Master@127.23.244.62:34127 shutting down...
I20240501 08:11:57.915581 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:11:57.915830 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:57.915925 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9e3f230e2f5c4705b790612eb398d55f: stopping tablet replica
I20240501 08:11:57.929080 24528 master.cc:552] Master@127.23.244.62:34127 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12665 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 08:11:57.939877 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:40291
I20240501 08:11:57.940339 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:57.942729 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:57.943581 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:57.944212 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:57.944869 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:57.945084 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:57.945132 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:57.945156 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551117945151 us; error 0 us; skew 500 ppm
I20240501 08:11:57.945302 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:57.946568 24528 webserver.cc:460] Webserver started at http://127.23.244.62:41965/ using document root <none> and password file <none>
I20240501 08:11:57.946770 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:57.946827 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:57.946945 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:57.947546 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "77cd410414974b6f95f751221b27c68d"
format_stamp: "Formatted at 2024-05-01 08:11:57 on dist-test-slave-fck4"
I20240501 08:11:57.950021 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:57.951470 27406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:57.951848 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:11:57.951942 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "77cd410414974b6f95f751221b27c68d"
format_stamp: "Formatted at 2024-05-01 08:11:57 on dist-test-slave-fck4"
I20240501 08:11:57.952054 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:57.961200 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:57.961745 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:57.971680 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:40291
I20240501 08:11:57.971699 27457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:40291 every 8 connection(s)
I20240501 08:11:57.972927 27458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.976665 27458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d: Bootstrap starting.
I20240501 08:11:57.977641 27458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.978932 27458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d: No bootstrap required, opened a new log
I20240501 08:11:57.979435 27458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER }
I20240501 08:11:57.979547 27458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.979660 27458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cd410414974b6f95f751221b27c68d, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.979892 27458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER }
I20240501 08:11:57.980026 27458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:57.980068 27458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:57.980113 27458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.981743 27458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER }
I20240501 08:11:57.981894 27458 leader_election.cc:302] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cd410414974b6f95f751221b27c68d; no voters:
I20240501 08:11:57.982190 27458 leader_election.cc:288] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:57.982308 27461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.982734 27461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 1 LEADER]: Becoming Leader. State: Replica: 77cd410414974b6f95f751221b27c68d, State: Running, Role: LEADER
I20240501 08:11:57.982975 27458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:57.982949 27461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER }
I20240501 08:11:57.983644 27463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cd410414974b6f95f751221b27c68d. Latest consensus state: current_term: 1 leader_uuid: "77cd410414974b6f95f751221b27c68d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER } }
I20240501 08:11:57.983713 27462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cd410414974b6f95f751221b27c68d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cd410414974b6f95f751221b27c68d" member_type: VOTER } }
I20240501 08:11:57.983800 27463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:57.983804 27462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:57.984850 27465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:57.986845 27465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:57.987944 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:11:57.989890 27465 catalog_manager.cc:1347] Generated new cluster ID: 9becb89a57e946eb9c65b7efa0eadac2
I20240501 08:11:57.989965 27465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:58.006736 27465 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:58.007493 27465 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:58.014626 27465 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d: Generated new TSK 0
I20240501 08:11:58.014832 27465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:58.020892 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:58.023492 27479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:58.024226 27480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:58.024461 27482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:58.025218 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:58.025435 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:58.025497 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:58.025530 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551118025526 us; error 0 us; skew 500 ppm
I20240501 08:11:58.025679 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:58.026895 24528 webserver.cc:460] Webserver started at http://127.23.244.1:33203/ using document root <none> and password file <none>
I20240501 08:11:58.027117 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:58.027180 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:58.027302 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:58.027943 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "090985d0089d4a0e8964843b12da2153"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.030395 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:58.031982 27487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.032300 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:11:58.032406 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "090985d0089d4a0e8964843b12da2153"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.032513 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:58.044061 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:58.044565 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:58.045116 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:58.045974 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:58.046028 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.046095 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:58.046136 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.059052 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:42429
I20240501 08:11:58.059067 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:42429 every 8 connection(s)
I20240501 08:11:58.060514 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:58.063313 27554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:58.064083 27555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:58.066749 24528 server_base.cc:1034] running on GCE node
W20240501 08:11:58.067011 27557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:58.067384 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:58.067445 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:58.067481 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551118067476 us; error 0 us; skew 500 ppm
I20240501 08:11:58.067611 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:58.068858 24528 webserver.cc:460] Webserver started at http://127.23.244.2:34627/ using document root <none> and password file <none>
I20240501 08:11:58.069103 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:58.069172 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:58.069270 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:58.069988 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "17050fe97c964f1785affcd1d2934983"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.070767 27550 heartbeater.cc:347] Connected to a master server at 127.23.244.62:40291
I20240501 08:11:58.070864 27550 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:58.071158 27550 heartbeater.cc:511] Master 127.23.244.62:40291 requested a full tablet report, sending...
I20240501 08:11:58.071914 27423 ts_manager.cc:194] Registered new tserver with Master: 090985d0089d4a0e8964843b12da2153 (127.23.244.1:42429)
I20240501 08:11:58.072949 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 08:11:58.073686 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42240
I20240501 08:11:58.075927 27562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.076241 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.003s sys 0.000s
I20240501 08:11:58.076333 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "17050fe97c964f1785affcd1d2934983"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.076447 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:58.098127 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:58.098655 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:58.099154 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:58.099964 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:58.100014 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.100064 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:58.100106 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.112579 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:41515
I20240501 08:11:58.112632 27624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:41515 every 8 connection(s)
I20240501 08:11:58.114037 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:58.116873 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:58.117985 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:58.118235 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:58.118886 24528 server_base.cc:1034] running on GCE node
I20240501 08:11:58.119163 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:11:58.119213 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:11:58.119236 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551118119232 us; error 0 us; skew 500 ppm
I20240501 08:11:58.119374 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:58.123234 24528 webserver.cc:460] Webserver started at http://127.23.244.3:38675/ using document root <none> and password file <none>
I20240501 08:11:58.123461 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:58.123554 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:58.123679 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:58.123826 27625 heartbeater.cc:347] Connected to a master server at 127.23.244.62:40291
I20240501 08:11:58.123956 27625 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:58.124261 27625 heartbeater.cc:511] Master 127.23.244.62:40291 requested a full tablet report, sending...
I20240501 08:11:58.124333 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.125036 27423 ts_manager.cc:194] Registered new tserver with Master: 17050fe97c964f1785affcd1d2934983 (127.23.244.2:41515)
I20240501 08:11:58.125722 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42256
I20240501 08:11:58.126562 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:58.127928 27636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.128252 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:11:58.128363 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4"
format_stamp: "Formatted at 2024-05-01 08:11:58 on dist-test-slave-fck4"
I20240501 08:11:58.128472 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:58.162874 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:58.163415 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:58.163969 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:58.164808 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:58.164882 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.164986 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:58.165047 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:58.177537 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:35315
I20240501 08:11:58.177558 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:35315 every 8 connection(s)
I20240501 08:11:58.184005 27699 heartbeater.cc:347] Connected to a master server at 127.23.244.62:40291
I20240501 08:11:58.184190 27699 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:58.184507 27699 heartbeater.cc:511] Master 127.23.244.62:40291 requested a full tablet report, sending...
I20240501 08:11:58.185439 27423 ts_manager.cc:194] Registered new tserver with Master: 26ca1e2dc51147bb93557dfcd3d5c7d4 (127.23.244.3:35315)
I20240501 08:11:58.185516 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007084636s
I20240501 08:11:58.186388 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42258
I20240501 08:11:59.075896 27550 heartbeater.cc:503] Master 127.23.244.62:40291 was elected leader, sending a full tablet report...
I20240501 08:11:59.126880 27625 heartbeater.cc:503] Master 127.23.244.62:40291 was elected leader, sending a full tablet report...
I20240501 08:11:59.187552 27699 heartbeater.cc:503] Master 127.23.244.62:40291 was elected leader, sending a full tablet report...
I20240501 08:11:59.218730 24528 test_util.cc:268] Using random seed: 174655508
I20240501 08:11:59.230981 27423 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42274:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:11:59.231501 27423 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:11:59.252553 27664 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6b7e82813e4db5bd0ec5213fc70715 (DEFAULT_TABLE table=test-workload [id=ae6f3420e96148dbb0f9cd3edd46650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.252858 27515 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6b7e82813e4db5bd0ec5213fc70715 (DEFAULT_TABLE table=test-workload [id=ae6f3420e96148dbb0f9cd3edd46650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.253099 27664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6b7e82813e4db5bd0ec5213fc70715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.253233 27515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6b7e82813e4db5bd0ec5213fc70715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.259588 27590 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6b7e82813e4db5bd0ec5213fc70715 (DEFAULT_TABLE table=test-workload [id=ae6f3420e96148dbb0f9cd3edd46650e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.259810 27720 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: Bootstrap starting.
I20240501 08:11:59.260033 27590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6b7e82813e4db5bd0ec5213fc70715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.260213 27721 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: Bootstrap starting.
I20240501 08:11:59.261435 27720 tablet_bootstrap.cc:654] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.261749 27721 tablet_bootstrap.cc:654] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.263433 27720 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: No bootstrap required, opened a new log
I20240501 08:11:59.263587 27720 ts_tablet_manager.cc:1367] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:59.263689 27721 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: No bootstrap required, opened a new log
I20240501 08:11:59.263836 27721 ts_tablet_manager.cc:1367] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:59.264302 27720 raft_consensus.cc:348] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.264374 27721 raft_consensus.cc:348] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.264503 27720 raft_consensus.cc:374] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.264535 27721 raft_consensus.cc:374] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.264628 27720 raft_consensus.cc:729] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090985d0089d4a0e8964843b12da2153, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.264652 27721 raft_consensus.cc:729] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ca1e2dc51147bb93557dfcd3d5c7d4, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.264923 27721 consensus_queue.cc:260] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.264905 27720 consensus_queue.cc:260] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.266245 27721 ts_tablet_manager.cc:1398] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:59.269032 27726 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: Bootstrap starting.
I20240501 08:11:59.270586 27726 tablet_bootstrap.cc:654] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.271674 27720 ts_tablet_manager.cc:1398] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240501 08:11:59.272258 27726 tablet_bootstrap.cc:492] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: No bootstrap required, opened a new log
I20240501 08:11:59.272432 27726 ts_tablet_manager.cc:1367] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:59.273185 27726 raft_consensus.cc:348] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.273391 27726 raft_consensus.cc:374] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.273475 27726 raft_consensus.cc:729] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17050fe97c964f1785affcd1d2934983, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.273700 27726 consensus_queue.cc:260] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.274333 27726 ts_tablet_manager.cc:1398] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:59.311244 27724 raft_consensus.cc:482] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.311484 27724 raft_consensus.cc:504] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.312268 27724 leader_election.cc:288] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17050fe97c964f1785affcd1d2934983 (127.23.244.2:41515), 090985d0089d4a0e8964843b12da2153 (127.23.244.1:42429)
I20240501 08:11:59.318707 27600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6b7e82813e4db5bd0ec5213fc70715" candidate_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17050fe97c964f1785affcd1d2934983" is_pre_election: true
I20240501 08:11:59.318984 27600 raft_consensus.cc:2454] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ca1e2dc51147bb93557dfcd3d5c7d4 in term 0.
I20240501 08:11:59.319419 27638 leader_election.cc:302] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17050fe97c964f1785affcd1d2934983, 26ca1e2dc51147bb93557dfcd3d5c7d4; no voters:
I20240501 08:11:59.319437 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6b7e82813e4db5bd0ec5213fc70715" candidate_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090985d0089d4a0e8964843b12da2153" is_pre_election: true
I20240501 08:11:59.319644 27525 raft_consensus.cc:2454] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ca1e2dc51147bb93557dfcd3d5c7d4 in term 0.
I20240501 08:11:59.319695 27724 raft_consensus.cc:2781] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.319797 27724 raft_consensus.cc:482] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.319890 27724 raft_consensus.cc:3037] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.321822 27724 raft_consensus.cc:504] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.322326 27724 leader_election.cc:288] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [CANDIDATE]: Term 1 election: Requested vote from peers 17050fe97c964f1785affcd1d2934983 (127.23.244.2:41515), 090985d0089d4a0e8964843b12da2153 (127.23.244.1:42429)
I20240501 08:11:59.322727 27600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6b7e82813e4db5bd0ec5213fc70715" candidate_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17050fe97c964f1785affcd1d2934983"
I20240501 08:11:59.322803 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6b7e82813e4db5bd0ec5213fc70715" candidate_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090985d0089d4a0e8964843b12da2153"
I20240501 08:11:59.322882 27600 raft_consensus.cc:3037] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.323009 27525 raft_consensus.cc:3037] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.324805 27600 raft_consensus.cc:2454] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ca1e2dc51147bb93557dfcd3d5c7d4 in term 1.
I20240501 08:11:59.324828 27525 raft_consensus.cc:2454] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ca1e2dc51147bb93557dfcd3d5c7d4 in term 1.
I20240501 08:11:59.325148 27638 leader_election.cc:302] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090985d0089d4a0e8964843b12da2153, 26ca1e2dc51147bb93557dfcd3d5c7d4; no voters:
I20240501 08:11:59.325527 27724 raft_consensus.cc:2781] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.325947 27724 raft_consensus.cc:686] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 1 LEADER]: Becoming Leader. State: Replica: 26ca1e2dc51147bb93557dfcd3d5c7d4, State: Running, Role: LEADER
I20240501 08:11:59.326203 27724 consensus_queue.cc:237] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } }
I20240501 08:11:59.328501 27423 catalog_manager.cc:5484] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ca1e2dc51147bb93557dfcd3d5c7d4 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ca1e2dc51147bb93557dfcd3d5c7d4" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 35315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:59.836082 27733 consensus_queue.cc:1035] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "090985d0089d4a0e8964843b12da2153" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.841080 27738 consensus_queue.cc:1035] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17050fe97c964f1785affcd1d2934983" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.987517 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:11:59.987920 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:11:59.988384 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:11:59.988482 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:00.990288 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:01.988871 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:01.989320 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:01.989467 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:01.992683 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:02.995064 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:03.989823 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:03.990257 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:03.990353 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:03.997520 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:05.000067 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:05.990713 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:05.991145 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:05.991276 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:06.002494 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:07.004904 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:07.991704 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:07.992146 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:07.992233 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:08.007197 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:09.009617 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:09.992614 27476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:09.993084 27476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:09.993181 27476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:10.011891 27471 placement_policy_util.cc:407] tablet 3c6b7e82813e4db5bd0ec5213fc70715: detected majority of replicas (2 of 3) at location L0
I20240501 08:12:10.392000 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:12:10.403342 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:10.403640 24528 tablet_replica.cc:330] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153: stopping tablet replica
I20240501 08:12:10.403821 24528 raft_consensus.cc:2229] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:10.404012 24528 raft_consensus.cc:2258] T 3c6b7e82813e4db5bd0ec5213fc70715 P 090985d0089d4a0e8964843b12da2153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.418298 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:12:10.421335 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:12:10.429352 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:10.429591 24528 tablet_replica.cc:330] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983: stopping tablet replica
I20240501 08:12:10.429787 24528 raft_consensus.cc:2229] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:10.429950 24528 raft_consensus.cc:2258] T 3c6b7e82813e4db5bd0ec5213fc70715 P 17050fe97c964f1785affcd1d2934983 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:10.433676 27638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.244.2:41515: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:12:10.434657 27638 consensus_peers.cc:485] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 -> Peer 17050fe97c964f1785affcd1d2934983 (127.23.244.2:41515): Couldn't send request to peer 17050fe97c964f1785affcd1d2934983. Status: Network error: Client connection negotiation failed: client connection to 127.23.244.2:41515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:10.442912 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:12:10.445552 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:12:10.454589 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:10.454818 24528 tablet_replica.cc:330] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4: stopping tablet replica
I20240501 08:12:10.454981 24528 raft_consensus.cc:2229] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:10.455227 24528 raft_consensus.cc:2258] T 3c6b7e82813e4db5bd0ec5213fc70715 P 26ca1e2dc51147bb93557dfcd3d5c7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.468354 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:12:10.471050 24528 master.cc:537] Master@127.23.244.62:40291 shutting down...
I20240501 08:12:10.478114 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:10.478323 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.478441 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 77cd410414974b6f95f751221b27c68d: stopping tablet replica
I20240501 08:12:10.491482 24528 master.cc:552] Master@127.23.244.62:40291 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12561 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 08:12:10.501747 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:38837
I20240501 08:12:10.502163 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:10.505014 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.505769 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.506292 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:10.507057 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:10.507304 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:10.507364 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:10.507408 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551130507389 us; error 0 us; skew 500 ppm
I20240501 08:12:10.507529 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:10.512616 24528 webserver.cc:460] Webserver started at http://127.23.244.62:42819/ using document root <none> and password file <none>
I20240501 08:12:10.512878 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:10.512946 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:10.513057 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:10.513646 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.515969 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:10.517482 27756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.517879 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:10.518002 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.518110 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:10.533102 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:10.540134 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:10.549719 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:38837
I20240501 08:12:10.549741 27807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:38837 every 8 connection(s)
I20240501 08:12:10.550961 27808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.554728 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d: Bootstrap starting.
I20240501 08:12:10.555631 27808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.556829 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d: No bootstrap required, opened a new log
I20240501 08:12:10.557315 27808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER }
I20240501 08:12:10.557422 27808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.557452 27808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfb0dd38cc6424a914275b0b8b4fb1d, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.557585 27808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER }
I20240501 08:12:10.557667 27808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:12:10.557701 27808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:12:10.557745 27808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.559305 27808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER }
I20240501 08:12:10.559450 27808 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfb0dd38cc6424a914275b0b8b4fb1d; no voters:
I20240501 08:12:10.559734 27808 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:12:10.559854 27811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.560271 27811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 1 LEADER]: Becoming Leader. State: Replica: 7cfb0dd38cc6424a914275b0b8b4fb1d, State: Running, Role: LEADER
I20240501 08:12:10.560412 27808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:12:10.560467 27811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER }
I20240501 08:12:10.561163 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfb0dd38cc6424a914275b0b8b4fb1d. Latest consensus state: current_term: 1 leader_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER } }
I20240501 08:12:10.561237 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb0dd38cc6424a914275b0b8b4fb1d" member_type: VOTER } }
I20240501 08:12:10.561328 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:10.561316 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:10.562709 27816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:12:10.564276 27816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:12:10.565243 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:12:10.567126 27816 catalog_manager.cc:1347] Generated new cluster ID: d697a1055e424942b264deecdc445927
I20240501 08:12:10.567200 27816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:12:10.578933 27816 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:12:10.579670 27816 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:12:10.589152 27816 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d: Generated new TSK 0
I20240501 08:12:10.589385 27816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:12:10.597823 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:10.600874 27829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.601482 27830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:10.602057 24528 server_base.cc:1034] running on GCE node
W20240501 08:12:10.602802 27832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:10.603091 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:10.603149 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:10.603201 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551130603192 us; error 0 us; skew 500 ppm
I20240501 08:12:10.603333 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:10.604597 24528 webserver.cc:460] Webserver started at http://127.23.244.1:33105/ using document root <none> and password file <none>
I20240501 08:12:10.604836 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:10.604899 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:10.605020 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:10.605671 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "a0210126bd0f40ab9fa38439bb737a0d"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.607944 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:12:10.609350 27837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.609618 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:10.609710 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "a0210126bd0f40ab9fa38439bb737a0d"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.609807 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:10.618402 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:10.618911 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:10.619354 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:10.620131 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:10.620183 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.620254 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:10.620287 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.632463 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:37903
I20240501 08:12:10.632486 27899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:37903 every 8 connection(s)
I20240501 08:12:10.633924 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:10.637251 27904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.637755 27907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.638504 27905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:10.639267 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:10.639467 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:10.639525 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:10.639568 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551130639564 us; error 0 us; skew 500 ppm
I20240501 08:12:10.639664 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:10.640292 27900 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:10.640400 27900 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:10.640779 27900 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:10.641403 24528 webserver.cc:460] Webserver started at http://127.23.244.2:40349/ using document root <none> and password file <none>
I20240501 08:12:10.641600 27773 ts_manager.cc:194] Registered new tserver with Master: a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:10.641600 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:10.641731 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:10.641842 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:10.642417 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "24ffb166dab84eb08b7578c6654deb3c"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.642665 27773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45430
I20240501 08:12:10.644588 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:10.645952 27912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.646239 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:10.646327 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "24ffb166dab84eb08b7578c6654deb3c"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.646415 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:10.653215 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:10.653614 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:10.654029 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:10.654670 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:10.654721 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.654785 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:10.654820 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.665732 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:46047
I20240501 08:12:10.665758 27974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:46047 every 8 connection(s)
I20240501 08:12:10.667011 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:10.670101 27978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.670738 27979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:10.671393 27981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:10.672019 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:10.672242 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:10.672327 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:10.672379 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551130672375 us; error 0 us; skew 500 ppm
I20240501 08:12:10.672538 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:10.672916 27975 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:10.673030 27975 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:10.673422 27975 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:10.673858 24528 webserver.cc:460] Webserver started at http://127.23.244.3:38107/ using document root <none> and password file <none>
I20240501 08:12:10.674144 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:10.674257 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:10.674290 27773 ts_manager.cc:194] Registered new tserver with Master: 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:10.674386 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:10.674939 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "2dec92348b7f41c9a5907ddb0f105129"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.675004 27773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45446
I20240501 08:12:10.677204 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 08:12:10.678543 27986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.678845 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:12:10.678931 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "2dec92348b7f41c9a5907ddb0f105129"
format_stamp: "Formatted at 2024-05-01 08:12:10 on dist-test-slave-fck4"
I20240501 08:12:10.679059 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:10.700198 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:10.700600 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:10.701110 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:10.701831 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:10.701886 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.701956 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:10.701989 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:10.712443 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:36343
I20240501 08:12:10.712476 28048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:36343 every 8 connection(s)
I20240501 08:12:10.718501 28049 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:10.718605 28049 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:10.718883 28049 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:10.719681 27773 ts_manager.cc:194] Registered new tserver with Master: 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:10.719974 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00676322s
I20240501 08:12:10.720368 27773 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45448
I20240501 08:12:11.643748 27900 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:11.676035 27975 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:11.721560 28049 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:11.747421 24528 test_util.cc:268] Using random seed: 187184213
I20240501 08:12:11.762195 27773 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45462:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:12:11.762952 27773 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:12:11.793390 28013 tablet_service.cc:1467] Processing CreateTablet for tablet 947f432452be499595d4472aeb4a6bbe (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:12:11.793759 28009 tablet_service.cc:1467] Processing CreateTablet for tablet df29fad2de5b4ac18e52d233ee63a6d0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:12:11.793929 28013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947f432452be499595d4472aeb4a6bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.794337 28009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df29fad2de5b4ac18e52d233ee63a6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.794757 28012 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5ff99f3f94039a7ba50580e56f0ff (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:12:11.794793 28014 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:12:11.795267 28011 tablet_service.cc:1467] Processing CreateTablet for tablet 9121766525b940d58127de0d7b9f5d57 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:12:11.795732 28010 tablet_service.cc:1467] Processing CreateTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:12:11.796097 28012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5ff99f3f94039a7ba50580e56f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.796134 27865 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:12:11.796412 28007 tablet_service.cc:1467] Processing CreateTablet for tablet 221cb4dc792649008eb83c64492ef820 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:12:11.796867 27857 tablet_service.cc:1467] Processing CreateTablet for tablet 25deaffcb0e441e590d7b300e1dcda69 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:12:11.797058 27864 tablet_service.cc:1467] Processing CreateTablet for tablet 947f432452be499595d4472aeb4a6bbe (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:12:11.797199 28005 tablet_service.cc:1467] Processing CreateTablet for tablet 9a506a24817641c8afbfe3f8a5e696c0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:12:11.796134 28008 tablet_service.cc:1467] Processing CreateTablet for tablet 40396f8dc9614e369ce570cf51305386 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:12:11.796535 27865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.796867 28006 tablet_service.cc:1467] Processing CreateTablet for tablet 25deaffcb0e441e590d7b300e1dcda69 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:12:11.797688 27863 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5ff99f3f94039a7ba50580e56f0ff (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:12:11.797842 28007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cb4dc792649008eb83c64492ef820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.798056 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5ff99f3f94039a7ba50580e56f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.801103 27862 tablet_service.cc:1467] Processing CreateTablet for tablet 9121766525b940d58127de0d7b9f5d57 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:12:11.801491 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9121766525b940d58127de0d7b9f5d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.803138 27940 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:12:11.803452 27857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25deaffcb0e441e590d7b300e1dcda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.803504 28014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.803862 27939 tablet_service.cc:1467] Processing CreateTablet for tablet 947f432452be499595d4472aeb4a6bbe (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:12:11.804548 27938 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5ff99f3f94039a7ba50580e56f0ff (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:12:11.805137 27861 tablet_service.cc:1467] Processing CreateTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:12:11.805181 27937 tablet_service.cc:1467] Processing CreateTablet for tablet 9121766525b940d58127de0d7b9f5d57 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:12:11.805531 27861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.805766 27936 tablet_service.cc:1467] Processing CreateTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:12:11.806403 27935 tablet_service.cc:1467] Processing CreateTablet for tablet df29fad2de5b4ac18e52d233ee63a6d0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:12:11.807075 27934 tablet_service.cc:1467] Processing CreateTablet for tablet 40396f8dc9614e369ce570cf51305386 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:12:11.807878 27933 tablet_service.cc:1467] Processing CreateTablet for tablet 221cb4dc792649008eb83c64492ef820 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:12:11.808466 27932 tablet_service.cc:1467] Processing CreateTablet for tablet 25deaffcb0e441e590d7b300e1dcda69 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:12:11.809013 28008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40396f8dc9614e369ce570cf51305386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.809141 27859 tablet_service.cc:1467] Processing CreateTablet for tablet 40396f8dc9614e369ce570cf51305386 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:12:11.809273 27931 tablet_service.cc:1467] Processing CreateTablet for tablet 9a506a24817641c8afbfe3f8a5e696c0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:12:11.809770 27858 tablet_service.cc:1467] Processing CreateTablet for tablet 221cb4dc792649008eb83c64492ef820 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:12:11.810210 27857 tablet_service.cc:1467] Processing CreateTablet for tablet 7af4b37f38284e8d9cdaaa56247b96e3 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:12:11.810484 27856 tablet_service.cc:1467] Processing CreateTablet for tablet 9a506a24817641c8afbfe3f8a5e696c0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:12:11.811079 27860 tablet_service.cc:1467] Processing CreateTablet for tablet df29fad2de5b4ac18e52d233ee63a6d0 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:12:11.811452 27864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947f432452be499595d4472aeb4a6bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.812968 28011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9121766525b940d58127de0d7b9f5d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.803510 27940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.817086 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25deaffcb0e441e590d7b300e1dcda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.820492 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40396f8dc9614e369ce570cf51305386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.825875 27863 tablet_service.cc:1467] Processing CreateTablet for tablet aa333a45f3a14263aba9557c7965c605 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:12:11.826211 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa333a45f3a14263aba9557c7965c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.828928 27931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a506a24817641c8afbfe3f8a5e696c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.829903 27933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cb4dc792649008eb83c64492ef820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.831166 27936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.831492 27939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947f432452be499595d4472aeb4a6bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.835688 27856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a506a24817641c8afbfe3f8a5e696c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.836827 27857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af4b37f38284e8d9cdaaa56247b96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.836874 27934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40396f8dc9614e369ce570cf51305386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.838186 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df29fad2de5b4ac18e52d233ee63a6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.838505 27858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cb4dc792649008eb83c64492ef820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.839038 27935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df29fad2de5b4ac18e52d233ee63a6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.840974 28010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.847121 28008 tablet_service.cc:1467] Processing CreateTablet for tablet aa333a45f3a14263aba9557c7965c605 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:12:11.848948 28005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a506a24817641c8afbfe3f8a5e696c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.849212 28008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa333a45f3a14263aba9557c7965c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.857870 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5ff99f3f94039a7ba50580e56f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.862612 27932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25deaffcb0e441e590d7b300e1dcda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.864780 27937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9121766525b940d58127de0d7b9f5d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.863164 28007 tablet_service.cc:1467] Processing CreateTablet for tablet 7af4b37f38284e8d9cdaaa56247b96e3 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:12:11.866276 28070 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.872927 27936 tablet_service.cc:1467] Processing CreateTablet for tablet 7af4b37f38284e8d9cdaaa56247b96e3 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:12:11.873442 27939 tablet_service.cc:1467] Processing CreateTablet for tablet aa333a45f3a14263aba9557c7965c605 (DEFAULT_TABLE table=test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:12:11.877089 28007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af4b37f38284e8d9cdaaa56247b96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.877928 28069 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.898047 27936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af4b37f38284e8d9cdaaa56247b96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.898387 27939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa333a45f3a14263aba9557c7965c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.908056 28069 tablet_bootstrap.cc:654] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.908427 28071 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:11.909731 28069 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.909765 28070 tablet_bootstrap.cc:654] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.909886 28069 ts_tablet_manager.cc:1367] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20240501 08:12:11.910369 28071 tablet_bootstrap.cc:654] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.910624 28069 raft_consensus.cc:348] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:11.910871 28069 raft_consensus.cc:374] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.910979 28069 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.911237 28069 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:11.911792 28070 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.911942 28070 ts_tablet_manager.cc:1367] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240501 08:12:11.912014 28069 ts_tablet_manager.cc:1398] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:12:11.912293 28069 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.912406 28071 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:11.912580 28071 ts_tablet_manager.cc:1367] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 08:12:11.912892 28070 raft_consensus.cc:348] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.913101 28070 raft_consensus.cc:374] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.913342 28071 raft_consensus.cc:348] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.913537 28071 raft_consensus.cc:374] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.913627 28071 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.913700 28070 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.913858 28071 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.913946 28070 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.914068 28069 tablet_bootstrap.cc:654] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.917652 28069 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.917804 28069 ts_tablet_manager.cc:1367] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:12:11.918542 28069 raft_consensus.cc:348] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.918756 28069 raft_consensus.cc:374] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.918845 28069 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.919100 28069 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.919618 28069 ts_tablet_manager.cc:1398] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:11.919879 28069 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.920974 28070 ts_tablet_manager.cc:1398] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 08:12:11.921303 28069 tablet_bootstrap.cc:654] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.923417 28071 ts_tablet_manager.cc:1398] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240501 08:12:11.923671 28071 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:11.925204 28071 tablet_bootstrap.cc:654] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.925771 28070 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.926050 28075 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.926187 28075 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:11.927312 28070 tablet_bootstrap.cc:654] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.928997 28075 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:11.929546 28070 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.929700 28070 ts_tablet_manager.cc:1367] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:11.930495 28070 raft_consensus.cc:348] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.930740 28070 raft_consensus.cc:374] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.930903 28070 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.932205 28070 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.932819 28070 ts_tablet_manager.cc:1398] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 08:12:11.934016 28070 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.935549 28070 tablet_bootstrap.cc:654] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.936898 28069 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.937031 28069 ts_tablet_manager.cc:1367] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20240501 08:12:11.937760 28069 raft_consensus.cc:348] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.937974 28069 raft_consensus.cc:374] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.938042 28069 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.938298 28069 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.938782 28069 ts_tablet_manager.cc:1398] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:11.939488 28069 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.941570 28069 tablet_bootstrap.cc:654] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.944298 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
W20240501 08:12:11.945785 27838 leader_election.cc:341] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.946338 28071 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:11.946520 28071 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20240501 08:12:11.947201 28071 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.947438 28071 raft_consensus.cc:374] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.947520 28071 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.947736 28071 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.948256 28071 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:11.948544 28071 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:11.950500 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:11.950879 28071 tablet_bootstrap.cc:654] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.951174 28070 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
W20240501 08:12:11.951232 27840 leader_election.cc:341] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.951388 27840 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0210126bd0f40ab9fa38439bb737a0d; no voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:11.951753 28075 raft_consensus.cc:2726] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:11.951385 28070 ts_tablet_manager.cc:1367] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 08:12:11.952870 28070 raft_consensus.cc:348] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.953161 28070 raft_consensus.cc:374] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.953263 28070 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.953498 28070 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.954082 28070 ts_tablet_manager.cc:1398] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:11.954353 28070 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.954515 28069 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.954690 28069 ts_tablet_manager.cc:1367] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20240501 08:12:11.955561 28069 raft_consensus.cc:348] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.955818 28069 raft_consensus.cc:374] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.955933 28070 tablet_bootstrap.cc:654] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.955909 28069 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.956285 28069 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.956883 28069 ts_tablet_manager.cc:1398] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:12:11.957136 28069 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.959026 28069 tablet_bootstrap.cc:654] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.959518 28071 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:11.959668 28071 ts_tablet_manager.cc:1367] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 08:12:11.960486 28071 raft_consensus.cc:348] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.960752 28071 raft_consensus.cc:374] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.960848 28071 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.961153 28071 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.963914 28070 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.963984 28071 ts_tablet_manager.cc:1398] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:11.964072 28070 ts_tablet_manager.cc:1367] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240501 08:12:11.964978 28071 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:11.965574 28070 raft_consensus.cc:348] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.965767 28070 raft_consensus.cc:374] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.965876 28070 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.966123 28070 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.966701 28070 ts_tablet_manager.cc:1398] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:11.966974 28070 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.967329 28071 tablet_bootstrap.cc:654] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.968511 28070 tablet_bootstrap.cc:654] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.972239 28071 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:11.972370 28071 ts_tablet_manager.cc:1367] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240501 08:12:11.973002 28071 raft_consensus.cc:348] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.973217 28071 raft_consensus.cc:374] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.973306 28071 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.973562 28071 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.973735 28069 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.973830 28070 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.973876 28069 ts_tablet_manager.cc:1367] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20240501 08:12:11.974000 28070 ts_tablet_manager.cc:1367] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 08:12:11.974586 28069 raft_consensus.cc:348] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.974783 28069 raft_consensus.cc:374] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.974805 28070 raft_consensus.cc:348] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.974939 28069 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.975010 28070 raft_consensus.cc:374] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.975104 28070 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.975157 28069 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.975317 28070 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.975739 28071 ts_tablet_manager.cc:1398] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:11.976011 28071 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:11.976173 28069 ts_tablet_manager.cc:1398] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:11.976413 28069 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.977938 28070 ts_tablet_manager.cc:1398] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:11.977993 28071 tablet_bootstrap.cc:654] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.978509 28069 tablet_bootstrap.cc:654] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.978960 28070 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.980681 28070 tablet_bootstrap.cc:654] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.984284 28071 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:11.984426 28071 ts_tablet_manager.cc:1367] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 08:12:11.984637 28069 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.984787 28069 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240501 08:12:11.985177 28071 raft_consensus.cc:348] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.985379 28070 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.985419 28071 raft_consensus.cc:374] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.985544 28071 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.985452 28069 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.985550 28070 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 08:12:11.985654 28069 raft_consensus.cc:374] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.985744 28069 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.985836 28071 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.985939 28069 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.986238 28070 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.986398 28069 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:11.986428 28070 raft_consensus.cc:374] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.986531 28070 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.986752 28069 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:11.986775 28070 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.988257 28076 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.988487 28069 tablet_bootstrap.cc:654] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.988425 28076 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.988749 28070 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:11.989025 28070 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.989634 28076 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:11.990531 28070 tablet_bootstrap.cc:654] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.992672 28075 raft_consensus.cc:482] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.992869 28075 raft_consensus.cc:504] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.993644 28075 leader_election.cc:288] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:11.994730 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
W20240501 08:12:11.995218 27838 leader_election.cc:341] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.994735 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
W20240501 08:12:11.995803 27840 leader_election.cc:341] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.995934 27840 leader_election.cc:302] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0210126bd0f40ab9fa38439bb737a0d; no voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:11.996332 28075 raft_consensus.cc:2726] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:11.997511 28070 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:11.997646 28070 ts_tablet_manager.cc:1367] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 08:12:11.998235 28070 raft_consensus.cc:348] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.998406 28070 raft_consensus.cc:374] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.998494 28070 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.998698 28069 tablet_bootstrap.cc:492] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:11.998696 28070 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.998893 28069 ts_tablet_manager.cc:1367] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240501 08:12:11.999331 28070 ts_tablet_manager.cc:1398] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:11.999621 28070 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:11.999747 28069 raft_consensus.cc:348] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:11.999994 28069 raft_consensus.cc:374] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.000089 28069 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.000384 28069 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.001247 28070 tablet_bootstrap.cc:654] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.002669 28069 ts_tablet_manager.cc:1398] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:12.003007 28069 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:12.004987 28069 tablet_bootstrap.cc:654] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.009040 28070 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:12.009074 28071 ts_tablet_manager.cc:1398] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240501 08:12:12.009182 28070 ts_tablet_manager.cc:1367] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 08:12:12.009464 28071 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.009944 28070 raft_consensus.cc:348] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.010103 28070 raft_consensus.cc:374] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.010183 28070 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.010411 28070 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.011566 28070 ts_tablet_manager.cc:1398] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.011852 28070 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:12.013967 28070 tablet_bootstrap.cc:654] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.014638 28071 tablet_bootstrap.cc:654] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.022581 28070 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:12.022766 28070 ts_tablet_manager.cc:1367] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240501 08:12:12.022930 28071 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.023133 28071 ts_tablet_manager.cc:1367] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240501 08:12:12.023605 28070 raft_consensus.cc:348] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.023851 28070 raft_consensus.cc:374] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.023947 28070 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.024087 28071 raft_consensus.cc:348] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.024219 28070 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.024389 28071 raft_consensus.cc:374] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.024533 28071 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.025714 28070 ts_tablet_manager.cc:1398] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:12.026019 28070 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:12.027789 28070 tablet_bootstrap.cc:654] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.029886 28071 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.031014 28070 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:12.031163 28070 ts_tablet_manager.cc:1367] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:12:12.031852 28070 raft_consensus.cc:348] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.032048 28070 raft_consensus.cc:374] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.032121 28070 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.032349 28070 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.033164 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.033510 28070 ts_tablet_manager.cc:1398] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.033536 27875 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.033766 28070 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:12.034025 27914 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.034351 28076 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.034472 28076 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.034581 28076 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.035383 28071 ts_tablet_manager.cc:1398] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240501 08:12:12.036396 28070 tablet_bootstrap.cc:654] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.038295 28071 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.038522 28076 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.039404 28076 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.039683 28075 raft_consensus.cc:482] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.039861 28075 raft_consensus.cc:504] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.040640 28075 leader_election.cc:288] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.041630 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.041918 27950 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 0.
I20240501 08:12:12.041630 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.042346 28024 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 0.
I20240501 08:12:12.042838 27838 leader_election.cc:302] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.043146 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.043408 28024 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.043467 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.043715 28023 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.044634 28070 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:12.044801 28070 ts_tablet_manager.cc:1367] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240501 08:12:12.045629 28071 tablet_bootstrap.cc:654] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.045572 28070 raft_consensus.cc:348] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.045759 28070 raft_consensus.cc:374] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.045838 28070 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.046085 28070 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.046777 28070 ts_tablet_manager.cc:1398] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:12.046799 28023 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.047073 28070 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Bootstrap starting.
I20240501 08:12:12.047145 27913 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.047240 28069 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:12.047386 28069 ts_tablet_manager.cc:1367] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.024s
I20240501 08:12:12.047447 28076 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.048476 28069 raft_consensus.cc:348] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.048694 28069 raft_consensus.cc:374] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.048825 28069 raft_consensus.cc:729] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.048944 28070 tablet_bootstrap.cc:654] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.049206 28069 consensus_queue.cc:260] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.052232 28075 raft_consensus.cc:2781] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.052366 28075 raft_consensus.cc:482] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.052472 28075 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.054419 28070 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: No bootstrap required, opened a new log
I20240501 08:12:12.054572 28070 ts_tablet_manager.cc:1367] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:12:12.055217 28070 raft_consensus.cc:348] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.055306 28075 raft_consensus.cc:504] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.055424 28070 raft_consensus.cc:374] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.055505 28070 raft_consensus.cc:729] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.055756 28070 consensus_queue.cc:260] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.056351 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.056558 27950 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.057247 28069 ts_tablet_manager.cc:1398] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20240501 08:12:12.057940 28069 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:12.058190 28070 ts_tablet_manager.cc:1398] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:12.059113 28077 raft_consensus.cc:482] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.059309 28077 raft_consensus.cc:504] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.061017 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.061211 27875 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.061708 27950 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 1.
I20240501 08:12:12.064978 28071 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.065652 27840 leader_election.cc:302] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.064921 28077 leader_election.cc:288] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.066452 28075 leader_election.cc:288] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.066797 28071 ts_tablet_manager.cc:1367] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.005s
I20240501 08:12:12.067831 28076 raft_consensus.cc:686] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:12.072866 28107 raft_consensus.cc:2781] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.073541 28071 raft_consensus.cc:348] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.073721 28071 raft_consensus.cc:374] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.073798 28071 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.074005 28071 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.074442 28071 ts_tablet_manager.cc:1398] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 08:12:12.068097 28076 consensus_queue.cc:237] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.076637 28071 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.077234 27875 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.076511 28107 raft_consensus.cc:686] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:12.078511 28107 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.078887 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.079697 28023 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.081472 28071 tablet_bootstrap.cc:654] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.084456 28023 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 1.
I20240501 08:12:12.086035 28069 tablet_bootstrap.cc:654] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.092206 27773 catalog_manager.cc:5484] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.104712 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.105003 27875 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.105594 27988 leader_election.cc:302] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.105599 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.105878 27950 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.105952 28077 raft_consensus.cc:2781] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.106060 28077 raft_consensus.cc:482] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.106164 28077 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.108680 28069 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:12.108836 28069 ts_tablet_manager.cc:1367] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.015s
I20240501 08:12:12.109082 28071 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.109239 28071 ts_tablet_manager.cc:1367] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.015s
I20240501 08:12:12.109665 28115 raft_consensus.cc:482] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.109480 28077 raft_consensus.cc:504] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.110034 28071 raft_consensus.cc:348] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.110219 28071 raft_consensus.cc:374] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.110124 28069 raft_consensus.cc:348] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.110296 28071 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.110327 28069 raft_consensus.cc:374] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.110450 28069 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.110524 28071 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.110652 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.110816 27875 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.110723 28069 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.111081 28071 ts_tablet_manager.cc:1398] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:12.111275 28069 ts_tablet_manager.cc:1398] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.111389 28071 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.111136 27772 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.111552 28069 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:12.113549 28071 tablet_bootstrap.cc:654] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.113795 28069 tablet_bootstrap.cc:654] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.113891 28115 raft_consensus.cc:504] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.115192 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.115434 27950 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.115936 27989 leader_election.cc:302] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.116015 27875 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.116101 27874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.116330 27874 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.116357 27988 leader_election.cc:302] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.117599 28077 leader_election.cc:288] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.117813 28077 raft_consensus.cc:2781] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.117906 28077 raft_consensus.cc:482] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.118008 28077 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.124644 28107 raft_consensus.cc:482] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.124856 28107 raft_consensus.cc:504] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.125795 28107 leader_election.cc:288] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.124699 28115 leader_election.cc:288] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.126279 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.126559 28023 raft_consensus.cc:2454] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 0.
I20240501 08:12:12.126641 28115 raft_consensus.cc:2781] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.127003 28115 raft_consensus.cc:686] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:12.127261 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.127789 28105 raft_consensus.cc:482] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 08:12:12.127883 27840 leader_election.cc:341] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:12.127928 28105 raft_consensus.cc:504] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.128229 27838 leader_election.cc:302] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters: 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:12.127728 28115 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.128605 28107 raft_consensus.cc:2781] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.128746 28107 raft_consensus.cc:482] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.128816 28105 leader_election.cc:288] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.128880 28107 raft_consensus.cc:3037] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.130291 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.130527 27875 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.130919 27914 leader_election.cc:302] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.131217 28105 raft_consensus.cc:2781] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.131357 28105 raft_consensus.cc:482] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.131471 28105 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.132499 28107 raft_consensus.cc:504] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.133924 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
W20240501 08:12:12.134531 27840 leader_election.cc:341] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:12.135629 28105 raft_consensus.cc:504] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.136519 28105 leader_election.cc:288] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.136561 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.136811 27875 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.138131 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.138352 27950 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.140523 27875 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.140997 27914 leader_election.cc:302] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.141326 28105 raft_consensus.cc:2781] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.141398 28107 leader_election.cc:288] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.141467 28105 raft_consensus.cc:686] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:12.141733 28105 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.141997 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.142139 28023 raft_consensus.cc:3037] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.143271 28077 raft_consensus.cc:504] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.144440 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.144690 27875 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.145725 28023 raft_consensus.cc:2454] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 1.
I20240501 08:12:12.146284 27838 leader_election.cc:302] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters: 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:12.147032 28077 leader_election.cc:288] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.147341 28107 raft_consensus.cc:2781] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.146975 27950 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.148150 28107 raft_consensus.cc:686] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:12.148542 28107 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.148926 27773 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.151947 27875 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.152474 27988 leader_election.cc:302] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.152843 28077 raft_consensus.cc:2781] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.153025 28077 raft_consensus.cc:686] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:12.153365 28077 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.153506 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.153777 28023 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.154150 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.154348 28024 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.156854 27773 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.157819 28024 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.158404 28071 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.158399 28069 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:12.158497 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.158610 28071 ts_tablet_manager.cc:1367] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.047s user 0.020s sys 0.025s
I20240501 08:12:12.158612 28069 ts_tablet_manager.cc:1367] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.015s
I20240501 08:12:12.158684 27950 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.159471 28069 raft_consensus.cc:348] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.159780 28069 raft_consensus.cc:374] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.159873 28069 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.160355 28071 raft_consensus.cc:348] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.160693 28071 raft_consensus.cc:374] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.160818 28071 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.161250 27950 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.161387 28069 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.162302 28069 ts_tablet_manager.cc:1398] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:12.161116 28071 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.163033 28071 ts_tablet_manager.cc:1398] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:12.163437 28071 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.163185 27772 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.165412 28069 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:12.165836 28071 tablet_bootstrap.cc:654] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.167048 28069 tablet_bootstrap.cc:654] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.171989 28071 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.171826 27773 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.172137 28071 ts_tablet_manager.cc:1367] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:12:12.172389 28069 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:12.172549 28069 ts_tablet_manager.cc:1367] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 08:12:12.172989 28071 raft_consensus.cc:348] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.173163 28071 raft_consensus.cc:374] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.173236 28071 raft_consensus.cc:729] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.173292 28069 raft_consensus.cc:348] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.173473 28069 raft_consensus.cc:374] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.173436 28071 consensus_queue.cc:260] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.173575 28069 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.173823 28071 ts_tablet_manager.cc:1398] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:12.173796 28069 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.174085 28071 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.174345 28069 ts_tablet_manager.cc:1398] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.174603 28069 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: Bootstrap starting.
I20240501 08:12:12.175488 28071 tablet_bootstrap.cc:654] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.176355 28069 tablet_bootstrap.cc:654] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.181833 28069 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: No bootstrap required, opened a new log
I20240501 08:12:12.181988 28069 ts_tablet_manager.cc:1367] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240501 08:12:12.182174 28071 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.182785 28071 ts_tablet_manager.cc:1367] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240501 08:12:12.182989 28069 raft_consensus.cc:348] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.183172 28069 raft_consensus.cc:374] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.183265 28069 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.183293 28071 raft_consensus.cc:348] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.183466 28071 raft_consensus.cc:374] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.183544 28071 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.183507 28069 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.183730 28071 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.184058 28069 ts_tablet_manager.cc:1398] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:12.184264 28071 ts_tablet_manager.cc:1398] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:12.184517 28071 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Bootstrap starting.
I20240501 08:12:12.185606 28071 tablet_bootstrap.cc:654] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.187572 28071 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: No bootstrap required, opened a new log
I20240501 08:12:12.187736 28071 ts_tablet_manager.cc:1367] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:12.188505 28071 raft_consensus.cc:348] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.188756 28071 raft_consensus.cc:374] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.188856 28071 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.189096 28071 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.189452 28071 ts_tablet_manager.cc:1398] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.202052 28107 raft_consensus.cc:482] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.202194 28107 raft_consensus.cc:504] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.202690 28107 leader_election.cc:288] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.203056 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.203083 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.203260 28118 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.203366 27950 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 0.
I20240501 08:12:12.203397 28024 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 0.
I20240501 08:12:12.203404 28118 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.203749 27840 leader_election.cc:302] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.203999 28107 raft_consensus.cc:2781] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.204114 28107 raft_consensus.cc:482] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.204208 28107 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.204604 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.204753 27875 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.205129 27988 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.205401 28077 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.205515 28077 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.205518 28118 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.205627 28077 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.205876 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.206044 27950 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.206710 28107 raft_consensus.cc:504] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.207150 28107 leader_election.cc:288] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.207540 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.207759 28077 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.207909 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.208066 27950 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.207760 28024 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.208917 28077 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.210335 27950 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 1.
I20240501 08:12:12.210582 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.210686 28024 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 1.
I20240501 08:12:12.210711 27875 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.211032 27840 leader_election.cc:302] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.211256 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.211410 27950 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.211483 28107 raft_consensus.cc:2781] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.211625 28107 raft_consensus.cc:686] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:12.211915 28107 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.213492 27875 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.213857 27988 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.213918 27950 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.214154 28077 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.214313 28105 raft_consensus.cc:482] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.214486 28105 raft_consensus.cc:504] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.215051 28105 leader_election.cc:288] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.214318 28077 raft_consensus.cc:686] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:12.214793 27771 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.215715 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.215788 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a506a24817641c8afbfe3f8a5e696c0" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.216053 28024 raft_consensus.cc:2379] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24ffb166dab84eb08b7578c6654deb3c in current term 1: Already voted for candidate a0210126bd0f40ab9fa38439bb737a0d in this term.
I20240501 08:12:12.216465 27913 leader_election.cc:302] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c; no voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:12.216743 28105 raft_consensus.cc:3037] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.218991 28105 raft_consensus.cc:2726] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:12.221545 28077 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.223912 27771 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.227484 28077 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.227609 28077 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.228291 28077 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.228600 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.228821 27875 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.228894 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.229074 27950 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.229213 27988 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.229445 28077 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.229527 28077 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.229604 28077 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.231165 28077 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.231660 28077 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.231997 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.231997 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.232197 27875 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.232197 27950 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.234753 27950 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.234753 27875 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.235134 27989 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.235464 28077 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.235584 28077 raft_consensus.cc:686] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:12.235788 28077 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.237941 27771 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.239403 28105 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.239531 28105 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.240072 28105 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:12.240447 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.240602 27875 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.240808 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.240964 27914 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.241009 28024 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.241189 28105 raft_consensus.cc:2781] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.241276 28105 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.241353 28105 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.242923 28105 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.243373 28105 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Requested vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:12.243716 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.243745 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.243865 28024 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.243868 27875 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.245491 27875 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.245512 28024 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.245781 27914 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:12.245998 28105 raft_consensus.cc:2781] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.246093 28105 raft_consensus.cc:686] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:12.246348 28105 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:12.248471 27771 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.319660 28105 raft_consensus.cc:482] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.319837 28105 raft_consensus.cc:504] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.320423 28105 leader_election.cc:288] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.320994 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.320996 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129" is_pre_election: true
I20240501 08:12:12.321206 28024 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.321208 27875 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 0.
I20240501 08:12:12.321578 27913 leader_election.cc:302] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.321799 28105 raft_consensus.cc:2781] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.321877 28105 raft_consensus.cc:482] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.321954 28105 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.323838 28105 raft_consensus.cc:504] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.324383 28105 leader_election.cc:288] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:12.324860 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.324862 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:12.325047 28024 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.325048 27875 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.327698 27875 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.327698 28024 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 1.
I20240501 08:12:12.328061 27913 leader_election.cc:302] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.328258 28105 raft_consensus.cc:2781] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.328356 28105 raft_consensus.cc:686] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:12.328563 28105 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.328769 28077 raft_consensus.cc:482] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.328912 28077 raft_consensus.cc:504] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.329627 28077 leader_election.cc:288] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.330041 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c" is_pre_election: true
I20240501 08:12:12.330041 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d" is_pre_election: true
I20240501 08:12:12.330291 27950 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.330296 27875 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 0.
I20240501 08:12:12.330641 27989 leader_election.cc:302] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.330848 28077 raft_consensus.cc:2781] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.330933 28077 raft_consensus.cc:482] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.331010 28077 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.331272 27771 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.332940 28077 raft_consensus.cc:504] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.333420 28077 leader_election.cc:288] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:12.333827 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:12.334004 27950 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.334100 27875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:12.334237 27875 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.336561 27950 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.336880 27989 leader_election.cc:302] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:12.337086 28077 raft_consensus.cc:2781] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.337179 28077 raft_consensus.cc:686] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:12.337381 27875 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 1.
I20240501 08:12:12.337386 28077 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:12.339500 27771 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.484447 27950 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.485780 28077 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.487627 27875 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.488374 28077 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.495478 28024 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.496127 27874 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.496212 27950 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.497001 28077 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.497563 28104 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.499687 27873 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.500617 28077 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.501780 27950 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:12.502358 28075 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.503566 28024 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.505440 28107 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240501 08:12:12.508975 27946 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.510957 28075 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.517549 28105 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.545667 27872 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.557555 28118 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
W20240501 08:12:12.556814 27838 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.244.3:36343, user_credentials={real_user=slave}} blocked reactor thread for 48622.5us
I20240501 08:12:12.559684 27948 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.559778 27947 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.562186 28077 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:12.559722 28075 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240501 08:12:12.564183 27945 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.565445 27871 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.562233 28118 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 08:12:12.571319 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240501 08:12:12.574232 28134 log.cc:924] Time spent T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240501 08:12:12.569816 28077 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 08:12:12.576606 28077 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.586831 28077 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240501 08:12:12.593469 27944 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:12.625564 28105 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:12.606897 28152 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.067s
W20240501 08:12:12.700515 28123 log.cc:924] Time spent T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
W20240501 08:12:12.700899 28120 log.cc:924] Time spent T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
W20240501 08:12:12.701800 28129 log.cc:924] Time spent T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
W20240501 08:12:12.702400 28112 log.cc:924] Time spent T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20240501 08:12:12.723584 28076 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
W20240501 08:12:12.745651 28159 log.cc:924] Time spent T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Append to log took a long time: real 0.192s user 0.000s sys 0.000s
I20240501 08:12:12.764030 28150 mvcc.cc:205] Tried to move back new op lower bound from 7022801438701854720 to 7022801437542617088. Current Snapshot: MvccSnapshot[applied={T|T < 7022801438695178240}]
I20240501 08:12:12.863328 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.866926 28076 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:12.874320 28111 log.cc:924] Time spent T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20240501 08:12:12.882118 28162 log.cc:924] Time spent T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20240501 08:12:12.924407 28157 log.cc:924] Time spent T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20240501 08:12:12.934154 28192 log.cc:924] Time spent T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20240501 08:12:12.938625 28200 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.939489 28200 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.940703 28200 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.977365 28021 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:12.977592 28021 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:12.978130 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:12.978904 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:12:13.008276 27913 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.244.3:36343, user_credentials={real_user=slave}} blocked reactor thread for 144985us
W20240501 08:12:13.012033 28191 log.cc:924] Time spent T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20240501 08:12:13.034258 28201 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20240501 08:12:13.179013 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.190088 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.214272 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.236297 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.256155 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.303457 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.303488 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.312016 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.330480 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.336133 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.338462 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.342907 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.351393 28000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.358729 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.358894 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.364883 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.379531 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.381666 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.384660 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.390941 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.393462 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.393476 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.397349 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.401666 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.403849 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.408152 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.409283 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.410435 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.415720 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.416761 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.418964 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.420017 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.427384 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.428393 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.430548 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.432538 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.436794 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.443214 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.443351 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.446465 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.450706 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.456367 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.457360 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.462797 28004 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
W20240501 08:12:13.463086 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Service unavailable: leader transfer in progress
I20240501 08:12:13.463194 28077 raft_consensus.cc:979] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:13.463317 28077 raft_consensus.cc:1067] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:13.463982 27875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38970
I20240501 08:12:13.464164 27875 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:13.464270 27875 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:13.466231 27875 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:13.466740 27875 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:13.467126 27946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:13.467211 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:13.467365 27946 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:13.467447 28021 raft_consensus.cc:3032] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:13.467567 28021 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:13.467833 28021 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:13.468143 28021 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:13.469494 27946 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:13.469899 27840 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:13.470235 28153 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:13.470391 28153 raft_consensus.cc:686] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:13.470611 28021 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:13.470659 28153 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:13.470952 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Illegal state: replica 2dec92348b7f41c9a5907ddb0f105129 is not leader of this config: current role FOLLOWER
I20240501 08:12:13.473062 27770 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 1 to 2, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:13.474514 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Illegal state: replica 2dec92348b7f41c9a5907ddb0f105129 is not leader of this config: current role FOLLOWER
W20240501 08:12:13.477862 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Illegal state: replica 2dec92348b7f41c9a5907ddb0f105129 is not leader of this config: current role FOLLOWER
W20240501 08:12:13.479930 28003 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46290: Illegal state: replica 2dec92348b7f41c9a5907ddb0f105129 is not leader of this config: current role FOLLOWER
I20240501 08:12:13.490175 28021 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:13.490213 27946 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:13.490716 28153 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:13.490923 28151 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:13.496568 28208 mvcc.cc:205] Tried to move back new op lower bound from 7022801442775236608 to 7022801442698780672. Current Snapshot: MvccSnapshot[applied={T|T < 7022801442771234816}]
I20240501 08:12:13.978782 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:13.985762 28213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:13.992622 28214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:13.993258 28216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:13.993863 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:13.994170 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:13.994233 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:13.994257 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551133994254 us; error 0 us; skew 500 ppm
I20240501 08:12:13.994396 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:13.995687 24528 webserver.cc:460] Webserver started at http://127.23.244.4:34855/ using document root <none> and password file <none>
I20240501 08:12:13.995913 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:13.995973 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:13.996100 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:13.996835 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-3-root/instance:
uuid: "e707ad1b8311480f9156e73e14b11059"
format_stamp: "Formatted at 2024-05-01 08:12:13 on dist-test-slave-fck4"
I20240501 08:12:13.999244 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:14.000753 28221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.001127 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:14.001222 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-3-root
uuid: "e707ad1b8311480f9156e73e14b11059"
format_stamp: "Formatted at 2024-05-01 08:12:13 on dist-test-slave-fck4"
I20240501 08:12:14.001343 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:14.017161 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:14.017652 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:14.018172 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:14.018996 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:14.019049 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.019112 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:14.019151 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.032667 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.4:39773
I20240501 08:12:14.032697 28283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.4:39773 every 8 connection(s)
I20240501 08:12:14.034252 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:14.037950 28288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:14.038471 28289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:14.039229 28291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:14.039937 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:14.040287 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:14.040360 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:14.040402 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551134040398 us; error 0 us; skew 500 ppm
I20240501 08:12:14.040539 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:14.040838 28284 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:14.040938 28284 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:14.041321 28284 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:14.041852 24528 webserver.cc:460] Webserver started at http://127.23.244.5:35337/ using document root <none> and password file <none>
I20240501 08:12:14.042055 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:14.042112 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:14.042201 27764 ts_manager.cc:194] Registered new tserver with Master: e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773)
I20240501 08:12:14.042235 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:14.042816 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-4-root/instance:
uuid: "892138e3535c4a72b0f0290a6b319d9b"
format_stamp: "Formatted at 2024-05-01 08:12:14 on dist-test-slave-fck4"
I20240501 08:12:14.042979 27764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48576
I20240501 08:12:14.045282 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:14.046674 28296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.047024 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 08:12:14.047118 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-4-root
uuid: "892138e3535c4a72b0f0290a6b319d9b"
format_stamp: "Formatted at 2024-05-01 08:12:14 on dist-test-slave-fck4"
I20240501 08:12:14.047200 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:14.063868 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:14.064334 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:14.064815 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:14.065591 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:14.065652 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.065706 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:14.065747 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.079424 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.5:38381
I20240501 08:12:14.079458 28358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.5:38381 every 8 connection(s)
I20240501 08:12:14.080984 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:14.084923 28362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:14.085950 28363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:14.086972 28365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:14.087618 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:14.088109 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:14.088215 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:14.088287 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551134088282 us; error 0 us; skew 500 ppm
I20240501 08:12:14.088433 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:14.089224 28359 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:14.089327 28359 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:14.089733 24528 webserver.cc:460] Webserver started at http://127.23.244.6:33795/ using document root <none> and password file <none>
I20240501 08:12:14.089752 28359 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:14.089956 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:14.090035 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:14.090170 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:14.090539 27764 ts_manager.cc:194] Registered new tserver with Master: 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:14.090785 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-5-root/instance:
uuid: "afccd612f3d743a2b54530e51e9a6a4f"
format_stamp: "Formatted at 2024-05-01 08:12:14 on dist-test-slave-fck4"
I20240501 08:12:14.091292 27764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48586
I20240501 08:12:14.093168 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:14.094574 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.094969 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:14.095067 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-5-root
uuid: "afccd612f3d743a2b54530e51e9a6a4f"
format_stamp: "Formatted at 2024-05-01 08:12:14 on dist-test-slave-fck4"
I20240501 08:12:14.095175 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:14.119632 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:14.120119 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:14.120642 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:14.121889 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:14.121968 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.122035 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:14.122082 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:14.135973 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.6:45837
I20240501 08:12:14.136036 28432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.6:45837 every 8 connection(s)
I20240501 08:12:14.142165 28433 heartbeater.cc:347] Connected to a master server at 127.23.244.62:38837
I20240501 08:12:14.142311 28433 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:14.142709 28433 heartbeater.cc:511] Master 127.23.244.62:38837 requested a full tablet report, sending...
I20240501 08:12:14.143463 27764 ts_manager.cc:194] Registered new tserver with Master: afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837)
I20240501 08:12:14.144127 27764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48590
I20240501 08:12:14.902763 27874 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:14.905225 27945 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:14.905707 28022 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:14.905778 28153 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.906164 28438 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.908129 28153 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:14.909108 27945 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:14.910796 28022 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
W20240501 08:12:14.913189 27840 consensus_peers.cc:485] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7af4b37f38284e8d9cdaaa56247b96e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.913899 27874 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:14.914011 27764 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:14.916699 27945 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:14.917280 28438 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.917706 28023 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:14.918160 28438 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.920754 28444 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.921528 28022 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.926054 27948 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.936322 27770 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index -1 to 26, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.938033 28022 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:14.940475 27874 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240501 08:12:14.940536 27838 consensus_peers.cc:485] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a506a24817641c8afbfe3f8a5e696c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.940965 28450 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.942006 27948 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:14.942806 28450 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.943349 28450 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.946149 27874 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.946375 27948 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
W20240501 08:12:14.948235 27989 consensus_peers.cc:485] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25deaffcb0e441e590d7b300e1dcda69. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.951593 27765 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.953989 27874 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:14.956394 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
W20240501 08:12:14.956863 27838 consensus_peers.cc:485] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 577a75e4e77d46f79a12ddcb864ebfae. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.956972 28438 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:14.957230 28022 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 08:12:14.957646 28438 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:14.959606 28444 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.960080 27948 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.961097 28022 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:14.965739 27874 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:14.967996 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:14.968412 28441 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.970291 27948 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:14.970755 28438 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:14.970594 27770 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 25, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
W20240501 08:12:14.974911 27840 consensus_peers.cc:485] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 221cb4dc792649008eb83c64492ef820. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.979352 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:14.985356 28444 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.985888 27945 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.986248 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:14.986380 28023 raft_consensus.cc:595] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:14.986800 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:14.986964 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:14.990706 28022 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:14.990998 27764 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index -1 to 26, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:15.004029 27945 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:15.006137 28022 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:15.006498 27874 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:15.006546 28205 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:15.006978 28458 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:15.014065 28458 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:15.016557 27874 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:15.016871 28022 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
W20240501 08:12:15.019830 27915 consensus_peers.cc:485] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9121766525b940d58127de0d7b9f5d57. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:15.019950 28453 raft_consensus.cc:979] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:15.020058 28453 raft_consensus.cc:1067] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:15.020601 27871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38970
I20240501 08:12:15.020778 27871 raft_consensus.cc:482] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:15.020853 27871 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:15.023288 27871 raft_consensus.cc:504] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:15.023288 27764 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 26, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:15.023917 27871 leader_election.cc:288] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:15.024451 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:15.024609 28022 raft_consensus.cc:3032] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:15.024600 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:15.024689 28022 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:15.024755 27948 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:15.024919 28022 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:15.025200 28022 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:15.027246 27948 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:15.027618 28022 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:15.027963 27838 leader_election.cc:302] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:15.028240 28444 raft_consensus.cc:2781] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:15.028406 28444 raft_consensus.cc:686] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:15.028661 28444 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:15.031211 27770 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 1 to 2, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:15.044004 28284 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:15.092279 28359 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:15.145180 28433 heartbeater.cc:503] Master 127.23.244.62:38837 was elected leader, sending a full tablet report...
I20240501 08:12:15.295500 28469 ts_tablet_manager.cc:902] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:15.297084 28469 tablet_copy_client.cc:308] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:15.302749 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 9a506a24817641c8afbfe3f8a5e696c0 from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:59638)
I20240501 08:12:15.302927 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 9a506a24817641c8afbfe3f8a5e696c0 from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:59638: session id = e707ad1b8311480f9156e73e14b11059-9a506a24817641c8afbfe3f8a5e696c0
I20240501 08:12:15.304793 27885 tablet_copy_source_session.cc:205] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.305891 28469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a506a24817641c8afbfe3f8a5e696c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.311060 28469 tablet_copy_client.cc:783] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.311336 28469 tablet_copy_client.cc:647] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.312872 28473 tablet_copy_client.cc:1002] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:15.323302 28475 ts_tablet_manager.cc:902] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:15.324955 28475 tablet_copy_client.cc:308] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:15.331005 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 7af4b37f38284e8d9cdaaa56247b96e3 from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:59648)
I20240501 08:12:15.331175 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 7af4b37f38284e8d9cdaaa56247b96e3 from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:59648: session id = 892138e3535c4a72b0f0290a6b319d9b-7af4b37f38284e8d9cdaaa56247b96e3
I20240501 08:12:15.333578 27885 tablet_copy_source_session.cc:205] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.334623 28475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af4b37f38284e8d9cdaaa56247b96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.339948 28475 tablet_copy_client.cc:783] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.340214 28475 tablet_copy_client.cc:647] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.341755 28479 tablet_copy_client.cc:1002] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:15.374857 28481 ts_tablet_manager.cc:902] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:15.376401 28481 tablet_copy_client.cc:308] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:15.377029 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 577a75e4e77d46f79a12ddcb864ebfae from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:59638)
I20240501 08:12:15.377213 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 577a75e4e77d46f79a12ddcb864ebfae from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:59638: session id = e707ad1b8311480f9156e73e14b11059-577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:15.379624 27885 tablet_copy_source_session.cc:205] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.380786 28481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.385576 28481 tablet_copy_client.cc:783] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.385892 28481 tablet_copy_client.cc:647] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.387378 28485 tablet_copy_client.cc:1002] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:15.393429 28486 ts_tablet_manager.cc:902] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:15.395071 28486 tablet_copy_client.cc:308] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.2:46047
I20240501 08:12:15.400593 27960 tablet_copy_service.cc:140] P 24ffb166dab84eb08b7578c6654deb3c: Received BeginTabletCopySession request for tablet 9121766525b940d58127de0d7b9f5d57 from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:42198)
I20240501 08:12:15.400776 27960 tablet_copy_service.cc:161] P 24ffb166dab84eb08b7578c6654deb3c: Beginning new tablet copy session on tablet 9121766525b940d58127de0d7b9f5d57 from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:42198: session id = 892138e3535c4a72b0f0290a6b319d9b-9121766525b940d58127de0d7b9f5d57
I20240501 08:12:15.403405 27960 tablet_copy_source_session.cc:205] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.404644 28486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9121766525b940d58127de0d7b9f5d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.409205 28486 tablet_copy_client.cc:783] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.409413 28486 tablet_copy_client.cc:647] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.411170 28490 tablet_copy_client.cc:1002] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:15.442580 28491 ts_tablet_manager.cc:902] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:15.450659 28491 tablet_copy_client.cc:308] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.3:36343
I20240501 08:12:15.457242 28034 tablet_copy_service.cc:140] P 2dec92348b7f41c9a5907ddb0f105129: Received BeginTabletCopySession request for tablet 25deaffcb0e441e590d7b300e1dcda69 from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:42710)
I20240501 08:12:15.457377 28034 tablet_copy_service.cc:161] P 2dec92348b7f41c9a5907ddb0f105129: Beginning new tablet copy session on tablet 25deaffcb0e441e590d7b300e1dcda69 from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:42710: session id = afccd612f3d743a2b54530e51e9a6a4f-25deaffcb0e441e590d7b300e1dcda69
I20240501 08:12:15.459189 28034 tablet_copy_source_session.cc:205] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.460341 28491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25deaffcb0e441e590d7b300e1dcda69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.466181 28491 tablet_copy_client.cc:783] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.466390 28491 tablet_copy_client.cc:647] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.467931 28495 tablet_copy_client.cc:1002] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:15.499027 27948 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:15.499464 28444 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:15.501518 28499 ts_tablet_manager.cc:902] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:15.505880 28499 tablet_copy_client.cc:308] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:15.511916 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 221cb4dc792649008eb83c64492ef820 from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:59660)
I20240501 08:12:15.512148 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 221cb4dc792649008eb83c64492ef820 from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:59660: session id = afccd612f3d743a2b54530e51e9a6a4f-221cb4dc792649008eb83c64492ef820
I20240501 08:12:15.514734 27885 tablet_copy_source_session.cc:205] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.515447 28022 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:15.515885 28499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cb4dc792649008eb83c64492ef820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.515846 28441 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:15.522145 28499 tablet_copy_client.cc:783] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.522404 28499 tablet_copy_client.cc:647] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.524122 28504 tablet_copy_client.cc:1002] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:16.987339 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:16.989886 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:16.990044 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:17.313503 28469 tablet_copy_client.cc:523] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.317183 28469 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
W20240501 08:12:17.331194 27840 consensus_peers.cc:485] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240501 08:12:17.342113 28475 tablet_copy_client.cc:523] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.344581 28475 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:17.351954 28469 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.352327 28469 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:17.352573 28469 ts_tablet_manager.cc:1367] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.001s
I20240501 08:12:17.353253 28469 raft_consensus.cc:348] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:17.353531 28469 raft_consensus.cc:729] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:17.353760 28469 consensus_queue.cc:260] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:17.354930 28469 ts_tablet_manager.cc:1398] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20240501 08:12:17.355626 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-9a506a24817641c8afbfe3f8a5e696c0 received from {username='slave'} at 127.0.0.1:59638
I20240501 08:12:17.355727 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session e707ad1b8311480f9156e73e14b11059-9a506a24817641c8afbfe3f8a5e696c0 on tablet 9a506a24817641c8afbfe3f8a5e696c0 with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:17.387761 28481 tablet_copy_client.cc:523] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.391320 28481 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
I20240501 08:12:17.391791 28475 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.392191 28475 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:17.392508 28475 ts_tablet_manager.cc:1367] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.048s user 0.033s sys 0.012s
I20240501 08:12:17.393206 28475 raft_consensus.cc:348] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:17.393409 28475 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:17.393646 28475 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:17.394387 28475 ts_tablet_manager.cc:1398] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:12:17.396528 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-7af4b37f38284e8d9cdaaa56247b96e3 received from {username='slave'} at 127.0.0.1:59648
I20240501 08:12:17.396621 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-7af4b37f38284e8d9cdaaa56247b96e3 on tablet 7af4b37f38284e8d9cdaaa56247b96e3 with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:17.411547 28486 tablet_copy_client.cc:523] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.414997 28486 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:17.429674 28481 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=93 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.430044 28481 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:17.430325 28481 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.039s user 0.035s sys 0.005s
I20240501 08:12:17.430958 28481 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:17.431236 28481 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:17.431453 28481 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:17.432051 28481 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:17.432610 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-577a75e4e77d46f79a12ddcb864ebfae received from {username='slave'} at 127.0.0.1:59638
I20240501 08:12:17.432705 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session e707ad1b8311480f9156e73e14b11059-577a75e4e77d46f79a12ddcb864ebfae on tablet 577a75e4e77d46f79a12ddcb864ebfae with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:17.453958 28486 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.454382 28486 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:17.454632 28486 ts_tablet_manager.cc:1367] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.007s
I20240501 08:12:17.455229 28486 raft_consensus.cc:348] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:17.455399 28486 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:17.455569 28486 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:17.456125 28486 ts_tablet_manager.cc:1398] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:17.456780 27960 tablet_copy_service.cc:342] P 24ffb166dab84eb08b7578c6654deb3c: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-9121766525b940d58127de0d7b9f5d57 received from {username='slave'} at 127.0.0.1:42198
I20240501 08:12:17.456890 27960 tablet_copy_service.cc:433] P 24ffb166dab84eb08b7578c6654deb3c: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-9121766525b940d58127de0d7b9f5d57 on tablet 9121766525b940d58127de0d7b9f5d57 with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:17.468284 28491 tablet_copy_client.cc:523] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.471827 28491 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:17.504669 28491 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.505151 28491 tablet_bootstrap.cc:492] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:17.505405 28491 ts_tablet_manager.cc:1367] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.003s
I20240501 08:12:17.506240 28259 raft_consensus.cc:1203] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 08:12:17.506156 28491 raft_consensus.cc:348] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:17.506465 28491 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:17.506669 28491 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:17.507452 28491 ts_tablet_manager.cc:1398] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:17.508644 28034 tablet_copy_service.cc:342] P 2dec92348b7f41c9a5907ddb0f105129: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-25deaffcb0e441e590d7b300e1dcda69 received from {username='slave'} at 127.0.0.1:42710
I20240501 08:12:17.508780 28034 tablet_copy_service.cc:433] P 2dec92348b7f41c9a5907ddb0f105129: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-25deaffcb0e441e590d7b300e1dcda69 on tablet 25deaffcb0e441e590d7b300e1dcda69 with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:17.524516 28499 tablet_copy_client.cc:523] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.527845 28499 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:17.562163 28499 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.562592 28499 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:17.562852 28499 ts_tablet_manager.cc:1367] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.009s
I20240501 08:12:17.563612 28499 raft_consensus.cc:348] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:17.563812 28499 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:17.564024 28499 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:17.564520 28499 ts_tablet_manager.cc:1398] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:17.565333 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-221cb4dc792649008eb83c64492ef820 received from {username='slave'} at 127.0.0.1:59660
I20240501 08:12:17.565428 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-221cb4dc792649008eb83c64492ef820 on tablet 221cb4dc792649008eb83c64492ef820 with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:17.612466 28408 raft_consensus.cc:1203] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 08:12:17.661485 28259 raft_consensus.cc:1203] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240501 08:12:17.845028 28334 raft_consensus.cc:1203] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 08:12:17.907366 28521 raft_consensus.cc:1050] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:17.908056 28521 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:17.910166 28259 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:17.910297 27945 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:17.910637 28023 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:17.910708 28438 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:17.910935 28522 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:17.911742 28438 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:17.915740 28438 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.917456 28259 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.919606 28023 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.921434 27945 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.922479 27770 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 26 to 27, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.932257 27874 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:17.933982 28259 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:17.934291 28023 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:17.935307 28438 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:17.935535 28522 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:17.937458 28522 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.938318 28259 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.938561 28023 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.943279 27764 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:17.943547 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a506a24817641c8afbfe3f8a5e696c0 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:17.946511 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a506a24817641c8afbfe3f8a5e696c0 with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:17.949074 28532 tablet_replica.cc:330] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:17.949235 28532 raft_consensus.cc:2229] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.949416 28532 raft_consensus.cc:2258] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.950217 28532 ts_tablet_manager.cc:1875] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:17.950284 28334 raft_consensus.cc:1203] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.25->[2.26-2.26] Dedup: 2.26->[]
I20240501 08:12:17.955670 28532 ts_tablet_manager.cc:1888] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 08:12:17.955778 28532 log.cc:1196] T 9a506a24817641c8afbfe3f8a5e696c0 P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/9a506a24817641c8afbfe3f8a5e696c0
I20240501 08:12:17.956465 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 9a506a24817641c8afbfe3f8a5e696c0 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:17.967486 28408 raft_consensus.cc:1203] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240501 08:12:18.108297 28521 raft_consensus.cc:1050] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:18.109195 28521 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:18.111212 27945 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:18.111469 28259 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:18.111824 28022 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:18.111907 28438 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:18.112279 28524 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:18.112548 28522 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:18.115725 28522 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.116520 28259 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.119732 27945 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.120443 28022 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.122503 27765 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 25 to 26, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.128561 27871 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:18.130136 28259 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.130136 27945 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.130661 28524 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.130914 28522 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.132928 28524 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.133788 27945 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.133816 28259 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.137526 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 577a75e4e77d46f79a12ddcb864ebfae with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:18.138527 27770 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index 26 to 27, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:18.141753 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:18.154526 28537 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:18.154726 28537 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.154978 28537 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.155817 28537 ts_tablet_manager.cc:1875] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:18.160909 28537 ts_tablet_manager.cc:1888] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240501 08:12:18.161046 28537 log.cc:1196] T 577a75e4e77d46f79a12ddcb864ebfae P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:18.161777 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet 577a75e4e77d46f79a12ddcb864ebfae (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:18.172389 28538 raft_consensus.cc:1050] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:18.173231 28538 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:18.175935 27945 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.176443 28511 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.176367 27874 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.177706 28408 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.178164 28539 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.180167 28539 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.185731 28539 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.186458 28408 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.187019 27945 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.191848 27874 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.192761 27764 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.198228 28023 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:18.200165 28408 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.200312 27945 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.200645 28539 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.200963 28511 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.203213 28541 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.203779 27945 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.203842 28408 raft_consensus.cc:2932] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.206777 27757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25deaffcb0e441e590d7b300e1dcda69 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:18.208245 27772 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.213063 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 25deaffcb0e441e590d7b300e1dcda69 with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:18.213797 28545 tablet_replica.cc:330] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:18.214038 28545 raft_consensus.cc:2229] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.214712 28545 raft_consensus.cc:2258] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.215500 28545 ts_tablet_manager.cc:1875] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:18.220939 28545 ts_tablet_manager.cc:1888] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 08:12:18.221032 28545 log.cc:1196] T 25deaffcb0e441e590d7b300e1dcda69 P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/25deaffcb0e441e590d7b300e1dcda69
I20240501 08:12:18.221791 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 25deaffcb0e441e590d7b300e1dcda69 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:18.301414 28547 raft_consensus.cc:1050] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:18.302160 28547 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:18.304774 28334 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.304842 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.305440 28547 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.305756 28509 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.306309 27874 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.308235 28547 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.318885 28546 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.319355 28023 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.319368 28334 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.324553 27764 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.324652 27874 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.330164 27945 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:18.331947 28334 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.332916 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.333380 28546 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.333789 28509 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.335727 28547 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.336318 28023 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.337141 28334 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.339787 27759 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9121766525b940d58127de0d7b9f5d57 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:18.341166 27770 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 27 to 28, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 1 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.345302 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 9121766525b940d58127de0d7b9f5d57 with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:18.345634 28545 tablet_replica.cc:330] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:18.345829 28545 raft_consensus.cc:2229] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.346115 28545 raft_consensus.cc:2258] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.347286 28545 ts_tablet_manager.cc:1875] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:18.352363 28545 ts_tablet_manager.cc:1888] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 08:12:18.352444 28545 log.cc:1196] T 9121766525b940d58127de0d7b9f5d57 P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/9121766525b940d58127de0d7b9f5d57
I20240501 08:12:18.353238 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 9121766525b940d58127de0d7b9f5d57 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:18.500566 28522 raft_consensus.cc:1050] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:18.501515 28522 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:18.503901 28022 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:18.503901 27945 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:18.503897 28334 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:18.504437 28438 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.504998 28553 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20240501 08:12:18.506255 28521 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.506839 28438 raft_consensus.cc:1050] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:18.507632 28438 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:18.510847 28553 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.511571 28022 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.513142 28334 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.517650 27764 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.518497 27945 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.521390 28408 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.521929 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.522825 28524 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.523250 27943 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:12:18.523670 27871 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:18.527056 28524 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.527366 28521 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:18.527510 27943 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:12:18.528100 28334 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:12:18.528627 28438 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.530350 28521 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.530786 28521 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.532804 27948 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.533085 28522 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.533838 28408 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.534642 27943 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.535123 28334 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:18.538144 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7af4b37f38284e8d9cdaaa56247b96e3 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:18.538967 27772 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 27 to 28, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.541617 28023 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.546200 27765 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: config changed from index 26 to 27, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.547224 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet 7af4b37f38284e8d9cdaaa56247b96e3 with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:18.547519 28537 tablet_replica.cc:330] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:18.547716 28537 raft_consensus.cc:2229] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.548009 28537 raft_consensus.cc:2258] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.549015 28537 ts_tablet_manager.cc:1875] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:18.552022 27871 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:18.554606 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.555039 28438 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.555199 28537 ts_tablet_manager.cc:1888] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.27
I20240501 08:12:18.555291 28537 log.cc:1196] T 7af4b37f38284e8d9cdaaa56247b96e3 P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/7af4b37f38284e8d9cdaaa56247b96e3
I20240501 08:12:18.555941 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet 7af4b37f38284e8d9cdaaa56247b96e3 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:18.556540 28408 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:12:18.557020 28438 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:18.557348 28522 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.557926 28023 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.561482 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 221cb4dc792649008eb83c64492ef820 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:18.561782 28408 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.562022 27770 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 27 to 28, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:18.564980 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cb4dc792649008eb83c64492ef820 with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:18.565603 28561 tablet_replica.cc:330] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:18.565779 28561 raft_consensus.cc:2229] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.565987 28561 raft_consensus.cc:2258] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.566810 28561 ts_tablet_manager.cc:1875] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:18.571411 28561 ts_tablet_manager.cc:1888] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240501 08:12:18.571498 28561 log.cc:1196] T 221cb4dc792649008eb83c64492ef820 P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/221cb4dc792649008eb83c64492ef820
I20240501 08:12:18.572094 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 221cb4dc792649008eb83c64492ef820 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:18.990418 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:19.004606 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:19.004817 27948 raft_consensus.cc:595] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:19.010587 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "221cb4dc792649008eb83c64492ef820"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:19.010771 27871 raft_consensus.cc:595] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Received request to transfer leadership to TS afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:19.011698 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:19.011873 28023 raft_consensus.cc:595] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:19.012791 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "e707ad1b8311480f9156e73e14b11059"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:19.012921 27871 raft_consensus.cc:595] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Received request to transfer leadership to TS e707ad1b8311480f9156e73e14b11059
I20240501 08:12:19.013659 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:19.013788 27871 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:19.014554 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "892138e3535c4a72b0f0290a6b319d9b"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:19.014698 27948 raft_consensus.cc:595] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Received request to transfer leadership to TS 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:19.015074 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240501 08:12:19.015298 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:19.065932 28565 raft_consensus.cc:979] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower afccd612f3d743a2b54530e51e9a6a4f to start an election
I20240501 08:12:19.066231 28566 raft_consensus.cc:1067] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Signalling peer afccd612f3d743a2b54530e51e9a6a4f to start an election
I20240501 08:12:19.066975 28408 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "221cb4dc792649008eb83c64492ef820"
dest_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
from {username='slave'} at 127.0.0.1:57210
I20240501 08:12:19.067222 28408 raft_consensus.cc:482] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.067324 28408 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.069263 28408 raft_consensus.cc:504] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:19.069921 28408 leader_election.cc:288] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [CANDIDATE]: Term 2 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:19.070295 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "afccd612f3d743a2b54530e51e9a6a4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:19.070376 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cb4dc792649008eb83c64492ef820" candidate_uuid: "afccd612f3d743a2b54530e51e9a6a4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:19.070489 28023 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.070489 27871 raft_consensus.cc:3032] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:19.070601 27871 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:19.070823 27871 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:19.071158 27871 raft_consensus.cc:3037] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.073302 28023 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afccd612f3d743a2b54530e51e9a6a4f in term 2.
I20240501 08:12:19.073746 27871 raft_consensus.cc:2454] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afccd612f3d743a2b54530e51e9a6a4f in term 2.
I20240501 08:12:19.073791 28565 raft_consensus.cc:979] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower e707ad1b8311480f9156e73e14b11059 to start an election
I20240501 08:12:19.073765 28371 leader_election.cc:302] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, afccd612f3d743a2b54530e51e9a6a4f; no voters:
I20240501 08:12:19.073912 28565 raft_consensus.cc:1067] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Signalling peer e707ad1b8311480f9156e73e14b11059 to start an election
I20240501 08:12:19.074433 28259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae"
dest_uuid: "e707ad1b8311480f9156e73e14b11059"
from {username='slave'} at 127.0.0.1:45742
I20240501 08:12:19.074455 28567 raft_consensus.cc:2781] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:19.074579 28259 raft_consensus.cc:482] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.074666 28259 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.074843 28567 raft_consensus.cc:686] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEADER]: Becoming Leader. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Running, Role: LEADER
I20240501 08:12:19.075102 28567 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:19.076668 28259 raft_consensus.cc:504] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:19.077369 28259 leader_election.cc:288] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [CANDIDATE]: Term 2 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:19.077224 27770 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: term changed from 1 to 2, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6). New cstate: current_term: 2 leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.077996 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "e707ad1b8311480f9156e73e14b11059" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:19.078147 27871 raft_consensus.cc:3032] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:19.078212 27871 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:19.078485 27871 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:19.078869 27871 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.081940 27871 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e707ad1b8311480f9156e73e14b11059 in term 2.
I20240501 08:12:19.082345 28223 leader_election.cc:302] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0210126bd0f40ab9fa38439bb737a0d, e707ad1b8311480f9156e73e14b11059; no voters:
I20240501 08:12:19.083757 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577a75e4e77d46f79a12ddcb864ebfae" candidate_uuid: "e707ad1b8311480f9156e73e14b11059" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:19.083904 27948 raft_consensus.cc:3037] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.085738 27948 raft_consensus.cc:2454] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e707ad1b8311480f9156e73e14b11059 in term 2.
I20240501 08:12:19.092988 28572 raft_consensus.cc:2781] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:19.093302 28572 raft_consensus.cc:686] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Becoming Leader. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Running, Role: LEADER
I20240501 08:12:19.093552 28572 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:19.095515 27770 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 reported cstate change: term changed from 1 to 2, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to e707ad1b8311480f9156e73e14b11059 (127.23.244.4). New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.111457 28566 raft_consensus.cc:979] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:19.111614 28566 raft_consensus.cc:1067] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:19.112246 27948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44008
I20240501 08:12:19.112411 27948 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.112478 27948 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:19.114379 27948 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.114869 27948 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:19.115309 28334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
I20240501 08:12:19.115309 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:19.115497 28334 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:19.115595 27871 raft_consensus.cc:3032] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:19.115656 27871 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:19.115875 27871 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.116156 27871 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:19.117399 28334 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:19.117784 27915 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 892138e3535c4a72b0f0290a6b319d9b; no voters:
I20240501 08:12:19.118019 28546 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:19.118085 27871 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:19.118520 28546 raft_consensus.cc:686] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:19.118760 28546 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.120991 27770 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 2 to 3, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.350512 28575 raft_consensus.cc:979] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:19.350673 28575 raft_consensus.cc:1067] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:19.351190 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46288
I20240501 08:12:19.351341 28023 raft_consensus.cc:482] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.351410 28023 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.353225 28023 raft_consensus.cc:504] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.353842 28023 leader_election.cc:288] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 2 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:19.354202 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:19.354223 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:19.354341 27948 raft_consensus.cc:3032] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:19.354385 27871 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.354404 27948 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:19.354638 27948 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.354869 27948 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.356401 27871 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 2.
I20240501 08:12:19.356590 27948 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 2.
I20240501 08:12:19.356779 27988 leader_election.cc:302] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:19.357041 28511 raft_consensus.cc:2781] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:19.357363 28511 raft_consensus.cc:686] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:19.357602 28511 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.359786 27770 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 1 to 2, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 2 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.371589 28576 raft_consensus.cc:979] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:19.371706 28576 raft_consensus.cc:1067] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:19.372274 27948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44018
I20240501 08:12:19.372416 27948 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.372502 27948 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.374357 27948 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.374873 27948 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 2 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:19.375247 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:19.375322 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:19.375418 27871 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.375490 28023 raft_consensus.cc:3032] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:19.375551 28023 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:19.375770 28023 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.376068 28023 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.377295 27871 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 2.
I20240501 08:12:19.377624 27914 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:19.377859 28575 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:19.377970 28575 raft_consensus.cc:686] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:19.378110 28023 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 2.
I20240501 08:12:19.378255 28575 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.380786 27770 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 1 to 2, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 2 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.419397 28546 raft_consensus.cc:979] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 892138e3535c4a72b0f0290a6b319d9b to start an election
I20240501 08:12:19.419561 28546 raft_consensus.cc:1067] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Signalling peer 892138e3535c4a72b0f0290a6b319d9b to start an election
I20240501 08:12:19.420079 28334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57"
dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
from {username='slave'} at 127.0.0.1:38416
I20240501 08:12:19.420276 28334 raft_consensus.cc:482] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:19.420392 28334 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.422575 28334 raft_consensus.cc:504] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.423812 28334 leader_election.cc:288] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [CANDIDATE]: Term 2 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:19.423851 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "892138e3535c4a72b0f0290a6b319d9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:19.424149 27948 raft_consensus.cc:3032] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:19.424216 27948 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:19.424428 27948 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.424657 27948 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.426939 27948 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892138e3535c4a72b0f0290a6b319d9b in term 2.
I20240501 08:12:19.427320 28299 leader_election.cc:302] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 892138e3535c4a72b0f0290a6b319d9b; no voters:
I20240501 08:12:19.428956 28581 raft_consensus.cc:2781] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:19.429102 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9121766525b940d58127de0d7b9f5d57" candidate_uuid: "892138e3535c4a72b0f0290a6b319d9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:19.429288 28023 raft_consensus.cc:3037] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:19.429325 28581 raft_consensus.cc:686] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEADER]: Becoming Leader. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Running, Role: LEADER
I20240501 08:12:19.429605 28581 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:19.431632 28023 raft_consensus.cc:2454] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892138e3535c4a72b0f0290a6b319d9b in term 2.
I20240501 08:12:19.432328 27770 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: term changed from 1 to 2, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5). New cstate: current_term: 2 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.462162 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:19.462716 28567 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:19.466529 27871 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:19.467231 28567 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:19.530000 28334 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:19.530597 28546 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:19.538002 27871 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:19.538627 28590 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 08:12:19.573917 27821 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 08:12:19.575593 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 21c5ff99f3f94039a7ba50580e56f0ff, TS 24ffb166dab84eb08b7578c6654deb3c -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:19.575682 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 21c5ff99f3f94039a7ba50580e56f0ff, TS 24ffb166dab84eb08b7578c6654deb3c -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:19.576984 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4e3e31e67e264e9f8f9d07dbc2d90f61, TS a0210126bd0f40ab9fa38439bb737a0d -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
W20240501 08:12:19.577065 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4e3e31e67e264e9f8f9d07dbc2d90f61, TS a0210126bd0f40ab9fa38439bb737a0d -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
W20240501 08:12:19.578325 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 40396f8dc9614e369ce570cf51305386, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.578331 27871 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240501 08:12:19.578420 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 40396f8dc9614e369ce570cf51305386, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.578821 28572 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 08:12:19.579663 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 25deaffcb0e441e590d7b300e1dcda69, TS 24ffb166dab84eb08b7578c6654deb3c -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
W20240501 08:12:19.579744 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 25deaffcb0e441e590d7b300e1dcda69, TS 24ffb166dab84eb08b7578c6654deb3c -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
W20240501 08:12:19.581739 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet df29fad2de5b4ac18e52d233ee63a6d0, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:19.581887 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet df29fad2de5b4ac18e52d233ee63a6d0, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:19.583256 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:12:19.583791 28572 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 08:12:19.594823 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 221cb4dc792649008eb83c64492ef820, TS a0210126bd0f40ab9fa38439bb737a0d -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
W20240501 08:12:19.594918 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 221cb4dc792649008eb83c64492ef820, TS a0210126bd0f40ab9fa38439bb737a0d -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:19.772112 27874 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:19.772666 28590 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:19.776185 28023 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:19.776629 28575 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:19.780869 27948 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:19.781324 28576 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:19.784971 27874 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:19.785400 28511 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:19.985955 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:19.986521 28581 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:19.990566 27948 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:19.991065 28604 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:20.600672 27874 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
W20240501 08:12:20.603117 27838 consensus_peers.cc:485] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df29fad2de5b4ac18e52d233ee63a6d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.603165 27948 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:20.603731 28023 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:20.603682 28610 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:20.604218 28602 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:20.606099 28610 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.606647 27948 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.610558 28022 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.610970 27765 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 27, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.611317 28023 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:20.620985 27874 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240501 08:12:20.621793 28511 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240501 08:12:20.624562 28576 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.625936 27874 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.629527 27948 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240501 08:12:20.630712 27948 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:20.632918 28608 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240501 08:12:20.635005 27764 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index -1 to 24, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.635246 27874 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:20.635820 28575 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:20.637975 28334 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:20.638509 28575 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 08:12:20.639298 27989 consensus_peers.cc:485] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 947f432452be499595d4472aeb4a6bbe. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.644299 28575 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 28 to 30, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:20.651683 27948 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.652057 28334 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:20.652057 27871 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 28 to 30, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:20.653654 27770 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index 28 to 30, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.660465 27874 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:20.663451 28259 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240501 08:12:20.663528 27840 consensus_peers.cc:485] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a506a24817641c8afbfe3f8a5e696c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.663938 28023 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:12:20.663939 28602 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:20.664870 28602 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 08:12:20.665994 27915 consensus_peers.cc:485] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7af4b37f38284e8d9cdaaa56247b96e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.667271 28615 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.667974 28259 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.668511 28023 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:20.672973 27764 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 28 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.680014 28334 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:20.681960 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:20.681960 27948 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:20.682581 28581 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:20.684499 28605 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.688238 27948 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.689832 27765 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 2 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.692909 28023 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:20.695245 27874 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:20.695847 27948 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:20.695832 28576 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240501 08:12:20.696394 27989 consensus_peers.cc:485] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21c5ff99f3f94039a7ba50580e56f0ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.696663 28576 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:20.696962 28581 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:20.698169 28023 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:20.699415 28576 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:20.700012 27874 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:20.699925 27948 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
W20240501 08:12:20.706197 28297 consensus_peers.cc:485] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9121766525b940d58127de0d7b9f5d57. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.706305 27770 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 2 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.006476 28636 ts_tablet_manager.cc:902] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:21.008292 28636 tablet_copy_client.cc:308] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:21.009069 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet df29fad2de5b4ac18e52d233ee63a6d0 from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:59638)
I20240501 08:12:21.009193 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet df29fad2de5b4ac18e52d233ee63a6d0 from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:59638: session id = e707ad1b8311480f9156e73e14b11059-df29fad2de5b4ac18e52d233ee63a6d0
I20240501 08:12:21.012104 27885 tablet_copy_source_session.cc:205] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.013403 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df29fad2de5b4ac18e52d233ee63a6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.015683 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:21.020829 28022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:21.021040 28022 raft_consensus.cc:595] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:21.021677 28636 tablet_copy_client.cc:783] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.022117 28636 tablet_copy_client.cc:647] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.022467 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:21.022696 27948 raft_consensus.cc:595] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:21.023754 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:21.023998 27948 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:21.024519 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 08:12:21.024919 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:21.026479 28640 tablet_copy_client.cc:1002] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:21.057265 28641 ts_tablet_manager.cc:902] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:21.058575 28641 tablet_copy_client.cc:308] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:21.059058 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 9a506a24817641c8afbfe3f8a5e696c0 from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:59648)
I20240501 08:12:21.059198 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 9a506a24817641c8afbfe3f8a5e696c0 from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:59648: session id = 892138e3535c4a72b0f0290a6b319d9b-9a506a24817641c8afbfe3f8a5e696c0
I20240501 08:12:21.061669 27885 tablet_copy_source_session.cc:205] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.062685 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a506a24817641c8afbfe3f8a5e696c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.067739 28641 tablet_copy_client.cc:783] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.067943 28641 tablet_copy_client.cc:647] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.069594 28645 tablet_copy_client.cc:1002] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:21.081023 28646 ts_tablet_manager.cc:902] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:21.082311 28646 tablet_copy_client.cc:308] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.3:36343
I20240501 08:12:21.082834 28034 tablet_copy_service.cc:140] P 2dec92348b7f41c9a5907ddb0f105129: Received BeginTabletCopySession request for tablet 21c5ff99f3f94039a7ba50580e56f0ff from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:42710)
I20240501 08:12:21.082957 28034 tablet_copy_service.cc:161] P 2dec92348b7f41c9a5907ddb0f105129: Beginning new tablet copy session on tablet 21c5ff99f3f94039a7ba50580e56f0ff from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:42710: session id = afccd612f3d743a2b54530e51e9a6a4f-21c5ff99f3f94039a7ba50580e56f0ff
I20240501 08:12:21.085453 28034 tablet_copy_source_session.cc:205] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.086443 28646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5ff99f3f94039a7ba50580e56f0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.091539 28646 tablet_copy_client.cc:783] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.091784 28646 tablet_copy_client.cc:647] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.093396 28650 tablet_copy_client.cc:1002] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:21.160784 28590 raft_consensus.cc:979] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:21.160954 28590 raft_consensus.cc:1067] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:21.161515 27874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38960
I20240501 08:12:21.161700 27874 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:21.161790 27874 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:21.164120 27874 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.165031 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:21.165166 27874 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 4 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:21.165202 27943 raft_consensus.cc:3032] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:12:21.165602 28334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
I20240501 08:12:21.165697 27943 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:21.165736 28334 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:21.165994 27943 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.166471 27943 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:21.168640 28334 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 4.
I20240501 08:12:21.169080 27840 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892138e3535c4a72b0f0290a6b319d9b, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:21.169092 27943 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 4.
I20240501 08:12:21.169406 28614 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:12:21.169553 28614 raft_consensus.cc:686] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:21.169833 28614 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.172348 27765 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 3 to 4, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 4 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.179126 28651 ts_tablet_manager.cc:902] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:21.181950 28651 tablet_copy_client.cc:308] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.3:36343
I20240501 08:12:21.182488 28034 tablet_copy_service.cc:140] P 2dec92348b7f41c9a5907ddb0f105129: Received BeginTabletCopySession request for tablet 947f432452be499595d4472aeb4a6bbe from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:42712)
I20240501 08:12:21.182638 28034 tablet_copy_service.cc:161] P 2dec92348b7f41c9a5907ddb0f105129: Beginning new tablet copy session on tablet 947f432452be499595d4472aeb4a6bbe from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:42712: session id = 892138e3535c4a72b0f0290a6b319d9b-947f432452be499595d4472aeb4a6bbe
I20240501 08:12:21.183266 27874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:21.183447 27874 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Received request to transfer leadership
I20240501 08:12:21.185040 28034 tablet_copy_source_session.cc:205] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.186096 28651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947f432452be499595d4472aeb4a6bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.191529 27874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:21.191747 27874 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Received request to transfer leadership
W20240501 08:12:21.192154 27758 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7af4b37f38284e8d9cdaaa56247b96e3 already in progress
I20240501 08:12:21.192330 28651 tablet_copy_client.cc:783] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.192651 28651 tablet_copy_client.cc:647] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.193215 27874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:21.193398 27874 raft_consensus.cc:595] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Received request to transfer leadership
W20240501 08:12:21.194058 27821 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7af4b37f38284e8d9cdaaa56247b96e3 already in progress
I20240501 08:12:21.194077 28655 tablet_copy_client.cc:1002] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
W20240501 08:12:21.194164 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7af4b37f38284e8d9cdaaa56247b96e3 already in progress
I20240501 08:12:21.264173 28608 raft_consensus.cc:979] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:21.264346 28608 raft_consensus.cc:1067] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:21.264961 27943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44018
I20240501 08:12:21.265134 27943 raft_consensus.cc:482] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:21.265234 27943 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.267632 27943 raft_consensus.cc:504] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.268265 27943 leader_election.cc:288] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:21.268651 27874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:21.268651 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:21.268891 27874 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.268909 28022 raft_consensus.cc:3032] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:21.268997 28022 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:21.269336 28022 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.269711 28022 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.271903 27874 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:21.272068 28022 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:21.272251 27914 leader_election.cc:302] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:21.272586 28590 raft_consensus.cc:2781] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:21.272737 28590 raft_consensus.cc:686] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:21.273000 28590 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.275975 27765 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 2 to 3, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.281970 28657 ts_tablet_manager.cc:902] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:21.283660 28657 tablet_copy_client.cc:308] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.5:38381
I20240501 08:12:21.289695 28344 tablet_copy_service.cc:140] P 892138e3535c4a72b0f0290a6b319d9b: Received BeginTabletCopySession request for tablet 9121766525b940d58127de0d7b9f5d57 from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:38436)
I20240501 08:12:21.289917 28344 tablet_copy_service.cc:161] P 892138e3535c4a72b0f0290a6b319d9b: Beginning new tablet copy session on tablet 9121766525b940d58127de0d7b9f5d57 from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:38436: session id = e707ad1b8311480f9156e73e14b11059-9121766525b940d58127de0d7b9f5d57
I20240501 08:12:21.292814 28344 tablet_copy_source_session.cc:205] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.293946 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9121766525b940d58127de0d7b9f5d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.299878 28657 tablet_copy_client.cc:783] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.300109 28657 tablet_copy_client.cc:647] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.301553 28661 tablet_copy_client.cc:1002] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:21.467770 28546 raft_consensus.cc:979] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:21.467957 28590 raft_consensus.cc:1067] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:21.468469 27871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38960
I20240501 08:12:21.468613 27871 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:21.468693 27871 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.470901 27871 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:21.471518 27871 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 3 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:21.471902 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:21.471958 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:21.472075 28022 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.472131 27948 raft_consensus.cc:3032] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:21.472203 27948 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:21.472448 27948 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:21.472852 27948 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:21.474774 28022 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 3.
I20240501 08:12:21.475082 27838 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:21.475355 27948 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 3.
I20240501 08:12:21.475430 28602 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:21.475562 28602 raft_consensus.cc:686] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:21.475932 28602 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:21.479655 27765 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 2 to 3, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 3 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:21.554513 27840 consensus_peers.cc:485] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7af4b37f38284e8d9cdaaa56247b96e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:21.595659 28334 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:21.596199 28664 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:21.598333 28602 raft_consensus.cc:979] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower 892138e3535c4a72b0f0290a6b319d9b to start an election
I20240501 08:12:21.598465 28602 raft_consensus.cc:1067] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Signalling peer 892138e3535c4a72b0f0290a6b319d9b to start an election
I20240501 08:12:21.599301 28334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3"
dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
from {username='slave'} at 127.0.0.1:38400
I20240501 08:12:21.599493 28334 raft_consensus.cc:482] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:21.599579 28334 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:21.602293 28334 raft_consensus.cc:504] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.603607 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "892138e3535c4a72b0f0290a6b319d9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:21.603834 27871 raft_consensus.cc:3032] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:12:21.603942 27871 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:21.604148 27943 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:21.604257 28334 leader_election.cc:288] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [CANDIDATE]: Term 5 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:21.604224 27871 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.604797 27871 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:21.604761 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af4b37f38284e8d9cdaaa56247b96e3" candidate_uuid: "892138e3535c4a72b0f0290a6b319d9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:21.604910 27943 raft_consensus.cc:3037] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:21.605748 28334 raft_consensus.cc:1226] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Rejecting Update request from peer a0210126bd0f40ab9fa38439bb737a0d for earlier term 4. Current term is 5. Ops: []
I20240501 08:12:21.607275 27871 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892138e3535c4a72b0f0290a6b319d9b in term 5.
I20240501 08:12:21.607770 27943 raft_consensus.cc:2454] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892138e3535c4a72b0f0290a6b319d9b in term 5.
I20240501 08:12:21.607757 28298 leader_election.cc:302] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892138e3535c4a72b0f0290a6b319d9b, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:21.608075 28605 raft_consensus.cc:2781] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:12:21.608206 28605 raft_consensus.cc:686] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 LEADER]: Becoming Leader. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Running, Role: LEADER
I20240501 08:12:21.608476 28605 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:21.611091 27765 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: term changed from 4 to 5, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5). New cstate: current_term: 5 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.683461 27871 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:21.683970 28590 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:21.688612 28022 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
W20240501 08:12:21.688910 27915 consensus_peers.cc:485] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:21.689270 28590 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:21.893579 28022 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 08:12:21.894039 28664 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:21.897328 27943 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 08:12:21.897822 28602 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:22.040632 27871 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240501 08:12:22.041213 28671 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:22.049191 27943 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240501 08:12:22.050411 28675 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
W20240501 08:12:22.056584 28299 consensus_peers.cc:485] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7af4b37f38284e8d9cdaaa56247b96e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:22.611696 28681 ts_tablet_manager.cc:902] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:22.613415 28681 tablet_copy_client.cc:308] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.5:38381
I20240501 08:12:22.627468 28344 tablet_copy_service.cc:140] P 892138e3535c4a72b0f0290a6b319d9b: Received BeginTabletCopySession request for tablet 7af4b37f38284e8d9cdaaa56247b96e3 from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:38444)
I20240501 08:12:22.627626 28344 tablet_copy_service.cc:161] P 892138e3535c4a72b0f0290a6b319d9b: Beginning new tablet copy session on tablet 7af4b37f38284e8d9cdaaa56247b96e3 from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:38444: session id = afccd612f3d743a2b54530e51e9a6a4f-7af4b37f38284e8d9cdaaa56247b96e3
I20240501 08:12:22.630106 28344 tablet_copy_source_session.cc:205] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:22.631199 28681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af4b37f38284e8d9cdaaa56247b96e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.637161 28681 tablet_copy_client.cc:783] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:22.637416 28681 tablet_copy_client.cc:647] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:22.638696 28685 tablet_copy_client.cc:1002] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:23.025244 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:23.026796 28636 tablet_copy_client.cc:523] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.029083 27943 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:23.029294 27943 raft_consensus.cc:595] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:23.030380 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:23.030552 27871 raft_consensus.cc:595] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:23.030928 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:12:23.031034 28636 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
I20240501 08:12:23.031229 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:23.061795 28636 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.062152 28636 tablet_bootstrap.cc:492] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:23.062383 28636 ts_tablet_manager.cc:1367] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.002s
I20240501 08:12:23.063027 28636 raft_consensus.cc:348] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.063222 28636 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:23.063432 28636 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.063901 28636 ts_tablet_manager.cc:1398] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:12:23.064848 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-df29fad2de5b4ac18e52d233ee63a6d0 received from {username='slave'} at 127.0.0.1:59638
I20240501 08:12:23.064942 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session e707ad1b8311480f9156e73e14b11059-df29fad2de5b4ac18e52d233ee63a6d0 on tablet df29fad2de5b4ac18e52d233ee63a6d0 with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:23.069936 28641 tablet_copy_client.cc:523] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.073212 28641 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:23.093744 28646 tablet_copy_client.cc:523] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.096818 28646 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:23.109694 28641 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.110039 28641 tablet_bootstrap.cc:492] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:23.110349 28641 ts_tablet_manager.cc:1367] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20240501 08:12:23.111089 28641 raft_consensus.cc:348] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:23.111310 28641 raft_consensus.cc:729] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:23.111495 28641 consensus_queue.cc:260] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:23.111984 28641 ts_tablet_manager.cc:1398] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:23.112895 28691 raft_consensus.cc:979] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:23.113063 28590 raft_consensus.cc:1067] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:23.113541 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-9a506a24817641c8afbfe3f8a5e696c0 received from {username='slave'} at 127.0.0.1:59648
I20240501 08:12:23.113644 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-9a506a24817641c8afbfe3f8a5e696c0 on tablet 9a506a24817641c8afbfe3f8a5e696c0 with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:23.113667 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46288
I20240501 08:12:23.113827 28023 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:23.113910 28023 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.116533 28023 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:23.117475 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:23.117643 27943 raft_consensus.cc:3032] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:23.117717 27943 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:23.117944 27943 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:23.118211 27943 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.118166 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:23.118324 27871 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.120074 27943 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 2.
I20240501 08:12:23.120508 27989 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:23.120707 28679 raft_consensus.cc:2781] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:23.120878 27871 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 2.
I20240501 08:12:23.121105 28023 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 2 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:23.122182 28679 raft_consensus.cc:686] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:23.122423 28679 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:23.125197 27765 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 1 to 2, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 2 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:23.144361 28646 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=85 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.144892 28646 tablet_bootstrap.cc:492] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:23.145182 28646 ts_tablet_manager.cc:1367] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.048s user 0.038s sys 0.011s
I20240501 08:12:23.145196 28699 raft_consensus.cc:979] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:23.145776 28699 raft_consensus.cc:1067] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:23.146013 28646 raft_consensus.cc:348] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:23.146250 28646 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:23.146471 28646 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:23.146523 27943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44008
I20240501 08:12:23.146817 27943 raft_consensus.cc:482] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:23.146925 27943 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:23.148036 28646 ts_tablet_manager.cc:1398] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:23.149627 27943 raft_consensus.cc:504] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.150183 27943 leader_election.cc:288] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:23.150653 27874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:23.150756 28034 tablet_copy_service.cc:342] P 2dec92348b7f41c9a5907ddb0f105129: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-21c5ff99f3f94039a7ba50580e56f0ff received from {username='slave'} at 127.0.0.1:42710
I20240501 08:12:23.150837 28034 tablet_copy_service.cc:433] P 2dec92348b7f41c9a5907ddb0f105129: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-21c5ff99f3f94039a7ba50580e56f0ff on tablet 21c5ff99f3f94039a7ba50580e56f0ff with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:23.150844 27874 raft_consensus.cc:3032] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:23.150954 27874 raft_consensus.cc:729] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:23.151268 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df29fad2de5b4ac18e52d233ee63a6d0" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:23.151180 27874 consensus_queue.cc:260] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.151458 28023 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:23.151688 27874 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:23.153769 28023 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:23.154125 27874 raft_consensus.cc:2454] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:23.154166 27913 leader_election.cc:302] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:23.154825 28691 raft_consensus.cc:2781] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:23.154950 28691 raft_consensus.cc:686] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:23.155239 28691 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.158313 27765 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 2 to 3, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:23.194576 28651 tablet_copy_client.cc:523] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.197845 28651 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:23.231881 28651 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.232316 28651 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:23.232605 28651 ts_tablet_manager.cc:1367] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20240501 08:12:23.233325 28651 raft_consensus.cc:348] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:23.233501 28651 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:23.233729 28651 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:23.235424 28651 ts_tablet_manager.cc:1398] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 08:12:23.236256 28034 tablet_copy_service.cc:342] P 2dec92348b7f41c9a5907ddb0f105129: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-947f432452be499595d4472aeb4a6bbe received from {username='slave'} at 127.0.0.1:42712
I20240501 08:12:23.236367 28034 tablet_copy_service.cc:433] P 2dec92348b7f41c9a5907ddb0f105129: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-947f432452be499595d4472aeb4a6bbe on tablet 947f432452be499595d4472aeb4a6bbe with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:23.300282 28408 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Advancing to term 3
I20240501 08:12:23.303766 28657 tablet_copy_client.cc:523] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.307550 28657 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
I20240501 08:12:23.349612 28657 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.350042 28657 tablet_bootstrap.cc:492] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:23.350306 28657 ts_tablet_manager.cc:1367] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.043s user 0.035s sys 0.005s
I20240501 08:12:23.350867 28657 raft_consensus.cc:348] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.351138 28657 raft_consensus.cc:729] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:23.351320 28657 consensus_queue.cc:260] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:23.351845 28657 ts_tablet_manager.cc:1398] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:23.352639 28344 tablet_copy_service.cc:342] P 892138e3535c4a72b0f0290a6b319d9b: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-9121766525b940d58127de0d7b9f5d57 received from {username='slave'} at 127.0.0.1:38436
I20240501 08:12:23.352770 28344 tablet_copy_service.cc:433] P 892138e3535c4a72b0f0290a6b319d9b: ending tablet copy session e707ad1b8311480f9156e73e14b11059-9121766525b940d58127de0d7b9f5d57 on tablet 9121766525b940d58127de0d7b9f5d57 with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:23.439036 28334 raft_consensus.cc:1203] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:12:23.545392 27943 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 08:12:23.545962 28679 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:23.551602 27871 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240501 08:12:23.552433 28692 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:23.577898 28334 raft_consensus.cc:1203] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[]
I20240501 08:12:23.596017 28259 raft_consensus.cc:3037] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Advancing to term 3
I20240501 08:12:23.599176 28259 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 LEARNER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:23.599756 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:23.602465 28691 raft_consensus.cc:1050] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
W20240501 08:12:23.602581 28691 raft_consensus.cc:1054] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: Unable to promote non-voter e707ad1b8311480f9156e73e14b11059: Illegal state: Leader has not yet committed an operation in its own term
I20240501 08:12:23.660195 28259 raft_consensus.cc:1203] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 08:12:23.689951 27874 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:23.690444 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:23.694856 28023 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:23.701176 28718 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:23.701685 28716 raft_consensus.cc:1050] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:23.702433 28716 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:23.704558 28259 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 LEARNER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.704964 27874 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.704964 28022 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.705511 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.706820 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.707020 28590 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.709528 28718 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.710263 27871 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.713512 28022 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.713985 28259 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.714613 27764 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index 27 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:23.731907 27943 consensus_queue.cc:237] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:23.733913 28259 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:23.734448 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:23.734714 27871 raft_consensus.cc:1261] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:23.735306 28691 consensus_queue.cc:1035] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:23.736519 28691 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.738586 27871 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.739534 27759 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df29fad2de5b4ac18e52d233ee63a6d0 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:23.740842 27770 catalog_manager.cc:5484] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:23.740845 28259 raft_consensus.cc:2932] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:23.748118 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet df29fad2de5b4ac18e52d233ee63a6d0 with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:23.748893 28724 tablet_replica.cc:330] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:23.749080 28724 raft_consensus.cc:2229] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:23.749346 28724 raft_consensus.cc:2258] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.750087 28724 ts_tablet_manager.cc:1875] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:23.754961 28724 ts_tablet_manager.cc:1888] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240501 08:12:23.755059 28724 log.cc:1196] T df29fad2de5b4ac18e52d233ee63a6d0 P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/df29fad2de5b4ac18e52d233ee63a6d0
I20240501 08:12:23.755781 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet df29fad2de5b4ac18e52d233ee63a6d0 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:23.765702 28719 raft_consensus.cc:1050] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:23.766515 28719 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:23.768530 28022 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.768849 27874 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.769038 28719 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.769258 28408 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 LEARNER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:23.769730 28719 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.770013 28691 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:23.773085 28716 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 27 to 29, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.774101 28408 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.774097 27871 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.779532 27770 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 27 to 29, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.780609 28022 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 27 to 29, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.786705 27943 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:23.788604 28022 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:23.788610 28408 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:23.789580 28691 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:23.789945 28590 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:23.792337 28718 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.793162 28022 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.795431 28408 raft_consensus.cc:2932] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:23.796816 27759 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21c5ff99f3f94039a7ba50580e56f0ff with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:23.798418 27765 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index 29 to 30, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:23.801928 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c5ff99f3f94039a7ba50580e56f0ff with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:23.805982 28728 tablet_replica.cc:330] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:23.806272 28728 raft_consensus.cc:2229] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:23.806579 28728 raft_consensus.cc:2258] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.807399 28728 ts_tablet_manager.cc:1875] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:23.812863 28728 ts_tablet_manager.cc:1888] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240501 08:12:23.812999 28728 log.cc:1196] T 21c5ff99f3f94039a7ba50580e56f0ff P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/21c5ff99f3f94039a7ba50580e56f0ff
I20240501 08:12:23.813668 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 21c5ff99f3f94039a7ba50580e56f0ff (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:24.003383 28730 raft_consensus.cc:1050] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:24.004277 28730 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:24.006558 28259 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:12:24.006943 28022 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:12:24.007115 28334 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:12:24.007547 28699 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:24.008247 28699 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:24.008817 28699 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:24.015515 28023 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.014917 28733 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.020465 27765 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.021267 28334 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.021908 28259 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.029932 27874 consensus_queue.cc:237] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:24.031576 28259 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:12:24.031729 28333 raft_consensus.cc:1261] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:12:24.031999 28733 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:24.032281 28699 consensus_queue.cc:1035] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:24.034871 28731 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.035418 28259 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.037550 28334 raft_consensus.cc:2932] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.038122 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a506a24817641c8afbfe3f8a5e696c0 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:24.039356 27765 catalog_manager.cc:5484] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 30 to 31, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 1 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:24.043464 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a506a24817641c8afbfe3f8a5e696c0 with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:24.043718 28724 tablet_replica.cc:330] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:24.043874 28724 raft_consensus.cc:2229] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.044106 28724 raft_consensus.cc:2258] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.044904 28724 ts_tablet_manager.cc:1875] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:24.049731 28724 ts_tablet_manager.cc:1888] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 08:12:24.049834 28724 log.cc:1196] T 9a506a24817641c8afbfe3f8a5e696c0 P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/9a506a24817641c8afbfe3f8a5e696c0
I20240501 08:12:24.050498 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet 9a506a24817641c8afbfe3f8a5e696c0 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:24.142918 28729 raft_consensus.cc:1050] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:24.143653 28729 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:24.145663 28333 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 LEARNER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 08:12:24.145671 27948 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 08:12:24.146242 27871 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240501 08:12:24.146222 28729 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:24.146530 28692 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:24.147186 28692 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240501 08:12:24.150405 28729 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.150907 27871 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.154434 27948 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.155014 27770 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 24 to 25, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.163708 28333 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.165907 28022 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:24.167650 28332 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:24.167853 27871 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240501 08:12:24.168314 28692 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:24.168541 28729 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:24.170928 28679 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.171437 28332 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.175302 27871 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.176141 27772 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 25 to 26, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 1 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:24.176750 27757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 947f432452be499595d4472aeb4a6bbe with cas_config_opid_index 25: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:24.181125 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 947f432452be499595d4472aeb4a6bbe with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 26) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:24.182101 28742 tablet_replica.cc:330] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:24.182281 28742 raft_consensus.cc:2229] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.182495 28742 raft_consensus.cc:2258] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.183723 28742 ts_tablet_manager.cc:1875] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:24.189278 28742 ts_tablet_manager.cc:1888] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.25
I20240501 08:12:24.189370 28742 log.cc:1196] T 947f432452be499595d4472aeb4a6bbe P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/947f432452be499595d4472aeb4a6bbe
I20240501 08:12:24.189960 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 947f432452be499595d4472aeb4a6bbe (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:24.272444 28743 raft_consensus.cc:1050] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:24.273324 28743 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:24.275211 28022 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:24.275406 27948 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:24.275521 28259 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:24.275898 28743 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:24.276187 28746 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:24.276563 28744 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:24.280140 28744 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.281021 27948 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.282270 28259 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.285203 27772 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:24.287389 28022 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.289680 28332 consensus_queue.cc:237] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:24.291954 28023 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:24.292510 28743 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:24.292690 28259 raft_consensus.cc:1261] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:24.293454 28743 consensus_queue.cc:1035] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:24.295065 28746 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.296121 28023 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.299022 27819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9121766525b940d58127de0d7b9f5d57 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:24.301384 27765 catalog_manager.cc:5484] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 31 to 32, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 2 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.301584 28259 raft_consensus.cc:2932] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:24.305068 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 9121766525b940d58127de0d7b9f5d57 with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:24.305344 28742 tablet_replica.cc:330] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:24.305521 28742 raft_consensus.cc:2229] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.305740 28742 raft_consensus.cc:2258] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.306511 28742 ts_tablet_manager.cc:1875] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:24.310637 28742 ts_tablet_manager.cc:1888] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 08:12:24.310724 28742 log.cc:1196] T 9121766525b940d58127de0d7b9f5d57 P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/9121766525b940d58127de0d7b9f5d57
I20240501 08:12:24.311268 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 9121766525b940d58127de0d7b9f5d57 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:24.639019 28681 tablet_copy_client.cc:523] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:24.641741 28681 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:24.669651 28681 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:24.670015 28681 tablet_bootstrap.cc:492] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:24.670253 28681 ts_tablet_manager.cc:1367] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.002s
I20240501 08:12:24.670796 28681 raft_consensus.cc:348] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:24.670975 28681 raft_consensus.cc:729] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:24.671150 28681 consensus_queue.cc:260] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:24.671588 28681 ts_tablet_manager.cc:1398] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:24.672251 28344 tablet_copy_service.cc:342] P 892138e3535c4a72b0f0290a6b319d9b: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-7af4b37f38284e8d9cdaaa56247b96e3 received from {username='slave'} at 127.0.0.1:38444
I20240501 08:12:24.672338 28344 tablet_copy_service.cc:433] P 892138e3535c4a72b0f0290a6b319d9b: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-7af4b37f38284e8d9cdaaa56247b96e3 on tablet 7af4b37f38284e8d9cdaaa56247b96e3 with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:25.031620 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:25.034364 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:25.034572 28023 raft_consensus.cc:595] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:25.035717 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:25.035873 27948 raft_consensus.cc:595] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:25.036303 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:12:25.036517 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:25.072584 28408 raft_consensus.cc:1203] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 LEARNER]: Deduplicated request from leader. Original: 4.31->[5.32-5.32] Dedup: 5.32->[]
I20240501 08:12:25.122303 28751 raft_consensus.cc:979] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:25.122499 28751 raft_consensus.cc:1067] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:25.123044 27871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38960
I20240501 08:12:25.123229 27871 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:25.123317 27871 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.125218 27871 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:25.125798 27871 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:25.126266 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:25.126272 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:25.126431 27948 raft_consensus.cc:3032] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:25.126446 28023 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.126514 27948 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:25.126739 27948 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:25.127033 27948 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.128711 28023 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:25.129174 27838 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:25.129252 27948 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:25.129398 28733 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:25.129851 28733 raft_consensus.cc:686] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:25.129932 28752 raft_consensus.cc:979] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:25.130056 28752 raft_consensus.cc:1067] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:25.130128 28733 consensus_queue.cc:237] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:25.130704 27871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38970
I20240501 08:12:25.130841 27871 raft_consensus.cc:482] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:25.130918 27871 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.132810 27772 catalog_manager.cc:5484] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 1 to 2, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.133025 27871 raft_consensus.cc:504] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:25.133720 27871 leader_election.cc:288] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Requested vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:25.133987 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:25.134191 28023 raft_consensus.cc:3032] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:25.134253 28023 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:25.134486 28023 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:25.134754 28023 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.136956 28333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
I20240501 08:12:25.137121 28333 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:25.138510 28023 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:25.138981 27838 leader_election.cc:302] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:25.139016 28333 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 2.
I20240501 08:12:25.139272 28733 raft_consensus.cc:2781] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:25.139425 28733 raft_consensus.cc:686] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:25.139688 28733 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:25.142585 27772 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 1 to 2, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.323189 28259 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:25.325835 27871 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:25.326316 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:25.326365 28761 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:25.326797 28607 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:25.329140 28761 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:25.329694 27871 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
W20240501 08:12:25.332640 28224 consensus_peers.cc:485] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 577a75e4e77d46f79a12ddcb864ebfae. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:25.333400 27948 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:25.335292 27765 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 27 to 29, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
W20240501 08:12:25.338871 27821 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID afccd612f3d743a2b54530e51e9a6a4f is already a member of the config. RaftConfig: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
W20240501 08:12:25.341919 27821 auto_rebalancer.cc:663] Could not move replica: Illegal state: 7af4b37f38284e8d9cdaaa56247b96e3: source replica 24ffb166dab84eb08b7578c6654deb3c does not have REPLACE attribute set
W20240501 08:12:25.341998 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 7af4b37f38284e8d9cdaaa56247b96e3: source replica 24ffb166dab84eb08b7578c6654deb3c does not have REPLACE attribute set
W20240501 08:12:25.344352 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4e3e31e67e264e9f8f9d07dbc2d90f61, TS a0210126bd0f40ab9fa38439bb737a0d -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
W20240501 08:12:25.344424 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4e3e31e67e264e9f8f9d07dbc2d90f61, TS a0210126bd0f40ab9fa38439bb737a0d -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
W20240501 08:12:25.346863 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aa333a45f3a14263aba9557c7965c605, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:25.346936 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aa333a45f3a14263aba9557c7965c605, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS 892138e3535c4a72b0f0290a6b319d9b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:25.349166 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 40396f8dc9614e369ce570cf51305386, TS 24ffb166dab84eb08b7578c6654deb3c -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:25.349236 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 40396f8dc9614e369ce570cf51305386, TS 24ffb166dab84eb08b7578c6654deb3c -> TS afccd612f3d743a2b54530e51e9a6a4f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
W20240501 08:12:25.351446 27821 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 221cb4dc792649008eb83c64492ef820, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
W20240501 08:12:25.351517 27821 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 221cb4dc792649008eb83c64492ef820, TS 2dec92348b7f41c9a5907ddb0f105129 -> TS e707ad1b8311480f9156e73e14b11059 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:25.499450 28767 raft_consensus.cc:1050] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:25.500301 28767 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:25.502926 27948 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240501 08:12:25.502926 28408 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 LEARNER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240501 08:12:25.503361 27871 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240501 08:12:25.503479 28768 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:25.503746 28767 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:25.504014 28744 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:25.508167 28744 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 30 to 33, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.508723 28408 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.512342 27871 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.512706 27948 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.513056 27770 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: config changed from index 30 to 33, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.520244 28333 consensus_queue.cc:237] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:25.521834 27948 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240501 08:12:25.521862 28408 raft_consensus.cc:1261] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 FOLLOWER]: Refusing update from remote peer 892138e3535c4a72b0f0290a6b319d9b: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240501 08:12:25.522408 28773 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:12:25.522858 28773 consensus_queue.cc:1035] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:12:25.524927 28773 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.525539 27948 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.525944 28408 raft_consensus.cc:2932] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:25.529407 27819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7af4b37f38284e8d9cdaaa56247b96e3 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:25.530067 27765 catalog_manager.cc:5484] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 33 to 34, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 5 leader_uuid: "892138e3535c4a72b0f0290a6b319d9b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:25.533409 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 7af4b37f38284e8d9cdaaa56247b96e3 with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:25.534189 28774 tablet_replica.cc:330] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:25.534420 28774 raft_consensus.cc:2229] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.535104 28774 raft_consensus.cc:2258] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.536511 28774 ts_tablet_manager.cc:1875] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:25.541457 28774 ts_tablet_manager.cc:1888] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.33
I20240501 08:12:25.541579 28774 log.cc:1196] T 7af4b37f38284e8d9cdaaa56247b96e3 P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/7af4b37f38284e8d9cdaaa56247b96e3
I20240501 08:12:25.542178 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 7af4b37f38284e8d9cdaaa56247b96e3 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:25.567173 28333 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:25.567665 28733 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:25.575106 28023 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 08:12:25.575683 28733 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:25.660789 27948 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:25.661424 28733 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:25.664853 28023 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 08:12:25.665621 28733 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:25.917344 28785 ts_tablet_manager.cc:902] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773)
I20240501 08:12:25.918812 28785 tablet_copy_client.cc:308] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.4:39773
I20240501 08:12:25.925508 28269 tablet_copy_service.cc:140] P e707ad1b8311480f9156e73e14b11059: Received BeginTabletCopySession request for tablet 577a75e4e77d46f79a12ddcb864ebfae from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:34944)
I20240501 08:12:25.925657 28269 tablet_copy_service.cc:161] P e707ad1b8311480f9156e73e14b11059: Beginning new tablet copy session on tablet 577a75e4e77d46f79a12ddcb864ebfae from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:34944: session id = afccd612f3d743a2b54530e51e9a6a4f-577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:25.928038 28269 tablet_copy_source_session.cc:205] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:25.929353 28785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.934968 28785 tablet_copy_client.cc:783] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:25.935192 28785 tablet_copy_client.cc:647] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:25.936569 28789 tablet_copy_client.cc:1002] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:27.036834 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:27.039712 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:27.039891 27871 raft_consensus.cc:595] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:27.040982 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:27.041163 27871 raft_consensus.cc:595] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:27.041504 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:12:27.041750 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:27.178006 28796 raft_consensus.cc:979] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:27.178189 28796 raft_consensus.cc:1067] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:27.178834 27948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44008
I20240501 08:12:27.179107 27948 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:27.179230 27948 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:27.181967 27948 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.182962 27948 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:27.183387 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:27.183496 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:27.183568 27871 raft_consensus.cc:3032] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:27.183614 28023 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:27.183634 27871 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:27.183856 27871 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.184193 27871 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:27.186264 28023 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:27.186583 27871 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:27.186697 27913 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:27.186990 28751 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:27.187347 28751 raft_consensus.cc:686] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:27.187570 28751 consensus_queue.cc:237] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.189971 27772 catalog_manager.cc:5484] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 2 to 3, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:27.349550 28733 raft_consensus.cc:979] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:27.349704 28733 raft_consensus.cc:1067] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:27.350231 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46272
I20240501 08:12:27.350441 28023 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:27.350589 28023 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:27.353323 28023 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.354691 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:27.354859 27871 raft_consensus.cc:3032] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:12:27.354964 27871 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:27.355204 27871 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.355458 27871 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:27.357697 27871 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:27.358079 28023 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:27.358044 27988 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:27.358389 28783 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:12:27.358525 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:27.358711 27948 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:27.360881 27948 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:27.361480 28783 raft_consensus.cc:686] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:27.361807 28783 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:27.365187 27772 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 3 to 4, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:27.589860 28023 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 08:12:27.590387 28751 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:27.595031 27871 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 08:12:27.595693 28807 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:27.780143 27948 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 08:12:27.780690 28783 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:27.784782 27871 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 08:12:27.786793 28783 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:27.936898 28785 tablet_copy_client.cc:523] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:27.940014 28785 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:27.972810 28785 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=93 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:27.973192 28785 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:27.973419 28785 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.033s user 0.032s sys 0.004s
I20240501 08:12:27.973984 28785 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:27.974218 28785 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:27.974385 28785 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:27.974892 28785 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:27.975451 28269 tablet_copy_service.cc:342] P e707ad1b8311480f9156e73e14b11059: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-577a75e4e77d46f79a12ddcb864ebfae received from {username='slave'} at 127.0.0.1:34944
I20240501 08:12:27.975551 28269 tablet_copy_service.cc:433] P e707ad1b8311480f9156e73e14b11059: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-577a75e4e77d46f79a12ddcb864ebfae on tablet 577a75e4e77d46f79a12ddcb864ebfae with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:28.378252 28408 raft_consensus.cc:1203] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240501 08:12:28.758229 28818 raft_consensus.cc:1050] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:28.759016 28818 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:28.761142 27871 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:28.761161 28408 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEARNER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:28.761601 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:28.762037 28795 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:28.762749 28795 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:28.763376 28795 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:28.765707 28820 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.766222 27871 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.770433 28408 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.770697 27770 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 29 to 30, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.772293 27948 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.779067 28259 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:28.780639 28408 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:28.780874 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:28.781113 28818 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:28.781641 28795 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:28.783193 28818 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.783803 28408 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.784358 27948 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:28.786957 27815 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 577a75e4e77d46f79a12ddcb864ebfae with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:28.788125 27772 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 30 to 31, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:28.791433 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:28.793072 28828 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:28.793329 28828 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.793605 28828 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.794258 28828 ts_tablet_manager.cc:1875] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:28.798399 28828 ts_tablet_manager.cc:1888] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240501 08:12:28.798483 28828 log.cc:1196] T 577a75e4e77d46f79a12ddcb864ebfae P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:28.799002 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 577a75e4e77d46f79a12ddcb864ebfae (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:29.042235 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:29.044951 27948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:29.045176 27948 raft_consensus.cc:595] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Received request to transfer leadership to TS afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:29.046042 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:29.046181 28023 raft_consensus.cc:595] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:29.046535 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:12:29.046751 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:29.191176 28829 raft_consensus.cc:979] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:29.191324 28829 raft_consensus.cc:1067] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:29.191840 27948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44018
I20240501 08:12:29.192001 27948 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:29.192085 27948 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:29.194151 27948 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:29.194672 27948 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Requested vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:29.194976 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:29.195031 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:29.195178 28023 raft_consensus.cc:3032] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:29.195230 27871 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:29.195238 28023 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:29.195505 28023 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:29.195744 28023 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:29.197281 27871 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:29.197678 27914 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:29.197777 28023 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 3.
I20240501 08:12:29.197903 28815 raft_consensus.cc:2781] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:29.198284 28815 raft_consensus.cc:686] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:29.198501 28815 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } }
I20240501 08:12:29.200466 27770 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 2 to 3, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.524950 28830 raft_consensus.cc:979] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower afccd612f3d743a2b54530e51e9a6a4f to start an election
I20240501 08:12:29.525085 28830 raft_consensus.cc:1067] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Signalling peer afccd612f3d743a2b54530e51e9a6a4f to start an election
I20240501 08:12:29.525635 28408 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff"
dest_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
from {username='slave'} at 127.0.0.1:57228
I20240501 08:12:29.525820 28408 raft_consensus.cc:482] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:29.525888 28408 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:29.527761 28408 raft_consensus.cc:504] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:29.528373 28408 leader_election.cc:288] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [CANDIDATE]: Term 4 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:29.528836 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "afccd612f3d743a2b54530e51e9a6a4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:29.529090 28023 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:29.530797 28023 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afccd612f3d743a2b54530e51e9a6a4f in term 4.
I20240501 08:12:29.531129 28371 leader_election.cc:302] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, afccd612f3d743a2b54530e51e9a6a4f; no voters:
I20240501 08:12:29.531373 28568 raft_consensus.cc:2781] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:12:29.531688 28568 raft_consensus.cc:686] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 4 LEADER]: Becoming Leader. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Running, Role: LEADER
I20240501 08:12:29.531926 28568 consensus_queue.cc:237] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:29.533543 27948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5ff99f3f94039a7ba50580e56f0ff" candidate_uuid: "afccd612f3d743a2b54530e51e9a6a4f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:29.533671 27948 raft_consensus.cc:3032] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:12:29.533735 27948 raft_consensus.cc:729] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:29.533926 27948 consensus_queue.cc:260] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:29.534178 27948 raft_consensus.cc:3037] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:29.534363 27770 catalog_manager.cc:5484] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: term changed from 3 to 4, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6). New cstate: current_term: 4 leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:29.535840 27948 raft_consensus.cc:2454] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afccd612f3d743a2b54530e51e9a6a4f in term 4.
I20240501 08:12:29.709869 28023 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 08:12:29.710371 28830 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:29.715094 27871 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240501 08:12:29.715708 28840 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 08:12:29.794787 28408 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:29.797245 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:29.797305 27871 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
W20240501 08:12:29.797576 28371 consensus_peers.cc:485] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 221cb4dc792649008eb83c64492ef820. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:29.797852 28568 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:29.798103 28834 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:29.800619 28568 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.801537 27871 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.805135 28023 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:29.804903 28022 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.805932 27765 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 28 to 30, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 2 leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
W20240501 08:12:29.808441 27989 consensus_peers.cc:485] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40396f8dc9614e369ce570cf51305386. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:29.809131 27874 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240501 08:12:29.810771 28829 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:29.813346 28829 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.813784 27871 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.818445 27948 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:29.818603 27943 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240501 08:12:29.819191 28829 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:29.820832 27765 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index -1 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.821230 27874 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240501 08:12:29.821409 28023 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
W20240501 08:12:29.821450 27915 consensus_peers.cc:485] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c -> Peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837): Couldn't send request to peer afccd612f3d743a2b54530e51e9a6a4f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e3e31e67e264e9f8f9d07dbc2d90f61. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:29.821695 28815 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:29.823189 28815 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:12:29.823194 27948 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.824278 28815 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:29.825214 27874 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
I20240501 08:12:29.829056 27770 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.830407 28258 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:29.833703 28023 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } } }
W20240501 08:12:29.834321 28224 consensus_peers.cc:485] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 -> Peer 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381): Couldn't send request to peer 892138e3535c4a72b0f0290a6b319d9b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 577a75e4e77d46f79a12ddcb864ebfae. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:29.837605 28408 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:29.837724 27948 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:29.838246 28854 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:29.838774 28854 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:29.840754 28855 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.841341 28408 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.844581 27765 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 31 to 32, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.845324 27874 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:29.846891 27948 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } } }
I20240501 08:12:29.848138 28023 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240501 08:12:29.848654 27838 consensus_peers.cc:485] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d -> Peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773): Couldn't send request to peer e707ad1b8311480f9156e73e14b11059. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 947f432452be499595d4472aeb4a6bbe. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:29.848932 28733 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:29.851042 28333 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:12:29.851523 28860 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:29.852535 28862 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.853089 28023 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.856961 27770 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 26 to 28, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.857546 28333 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, NON_VOTER e707ad1b8311480f9156e73e14b11059 (127.23.244.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } } }
I20240501 08:12:29.933631 28023 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:29.934252 28568 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:29.939524 27948 raft_consensus.cc:1261] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:29.940708 28568 consensus_queue.cc:1035] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:30.220594 28867 ts_tablet_manager.cc:902] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837)
I20240501 08:12:30.222189 28867 tablet_copy_client.cc:308] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.6:45837
I20240501 08:12:30.222790 28418 tablet_copy_service.cc:140] P afccd612f3d743a2b54530e51e9a6a4f: Received BeginTabletCopySession request for tablet 221cb4dc792649008eb83c64492ef820 from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:58454)
I20240501 08:12:30.222950 28418 tablet_copy_service.cc:161] P afccd612f3d743a2b54530e51e9a6a4f: Beginning new tablet copy session on tablet 221cb4dc792649008eb83c64492ef820 from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:58454: session id = e707ad1b8311480f9156e73e14b11059-221cb4dc792649008eb83c64492ef820
I20240501 08:12:30.225237 28418 tablet_copy_source_session.cc:205] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:30.226361 28867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cb4dc792649008eb83c64492ef820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.231379 28867 tablet_copy_client.cc:783] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:30.231638 28867 tablet_copy_client.cc:647] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:30.233000 28871 tablet_copy_client.cc:1002] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:30.236268 28872 ts_tablet_manager.cc:902] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:30.237506 28872 tablet_copy_client.cc:308] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.3:36343
I20240501 08:12:30.238055 28034 tablet_copy_service.cc:140] P 2dec92348b7f41c9a5907ddb0f105129: Received BeginTabletCopySession request for tablet 40396f8dc9614e369ce570cf51305386 from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:42712)
I20240501 08:12:30.238200 28034 tablet_copy_service.cc:161] P 2dec92348b7f41c9a5907ddb0f105129: Beginning new tablet copy session on tablet 40396f8dc9614e369ce570cf51305386 from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:42712: session id = 892138e3535c4a72b0f0290a6b319d9b-40396f8dc9614e369ce570cf51305386
I20240501 08:12:30.240447 28034 tablet_copy_source_session.cc:205] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:30.241683 28872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40396f8dc9614e369ce570cf51305386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.246629 28872 tablet_copy_client.cc:783] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:30.246917 28872 tablet_copy_client.cc:647] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:30.248520 28876 tablet_copy_client.cc:1002] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:30.252408 28877 ts_tablet_manager.cc:902] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Initiating tablet copy from peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:30.253937 28877 tablet_copy_client.cc:308] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.2:46047
I20240501 08:12:30.254630 27960 tablet_copy_service.cc:140] P 24ffb166dab84eb08b7578c6654deb3c: Received BeginTabletCopySession request for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 from peer afccd612f3d743a2b54530e51e9a6a4f ({username='slave'} at 127.0.0.1:54262)
I20240501 08:12:30.254786 27960 tablet_copy_service.cc:161] P 24ffb166dab84eb08b7578c6654deb3c: Beginning new tablet copy session on tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 from peer afccd612f3d743a2b54530e51e9a6a4f at {username='slave'} at 127.0.0.1:54262: session id = afccd612f3d743a2b54530e51e9a6a4f-4e3e31e67e264e9f8f9d07dbc2d90f61
I20240501 08:12:30.257400 27960 tablet_copy_source_session.cc:205] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:30.258634 28877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.264065 28877 tablet_copy_client.cc:783] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:30.264312 28877 tablet_copy_client.cc:647] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:30.265892 28881 tablet_copy_client.cc:1002] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:30.275681 28882 ts_tablet_manager.cc:902] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Initiating tablet copy from peer a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903)
I20240501 08:12:30.277398 28882 tablet_copy_client.cc:308] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.1:37903
I20240501 08:12:30.277967 27885 tablet_copy_service.cc:140] P a0210126bd0f40ab9fa38439bb737a0d: Received BeginTabletCopySession request for tablet 947f432452be499595d4472aeb4a6bbe from peer e707ad1b8311480f9156e73e14b11059 ({username='slave'} at 127.0.0.1:59638)
I20240501 08:12:30.278101 27885 tablet_copy_service.cc:161] P a0210126bd0f40ab9fa38439bb737a0d: Beginning new tablet copy session on tablet 947f432452be499595d4472aeb4a6bbe from peer e707ad1b8311480f9156e73e14b11059 at {username='slave'} at 127.0.0.1:59638: session id = e707ad1b8311480f9156e73e14b11059-947f432452be499595d4472aeb4a6bbe
I20240501 08:12:30.280938 27885 tablet_copy_source_session.cc:205] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:30.282011 28882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947f432452be499595d4472aeb4a6bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.287299 28882 tablet_copy_client.cc:783] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:30.287525 28882 tablet_copy_client.cc:647] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:30.288914 28886 tablet_copy_client.cc:1002] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:30.373052 28888 ts_tablet_manager.cc:902] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Initiating tablet copy from peer e707ad1b8311480f9156e73e14b11059 (127.23.244.4:39773)
I20240501 08:12:30.374492 28888 tablet_copy_client.cc:308] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Beginning tablet copy session from remote peer at address 127.23.244.4:39773
I20240501 08:12:30.375015 28269 tablet_copy_service.cc:140] P e707ad1b8311480f9156e73e14b11059: Received BeginTabletCopySession request for tablet 577a75e4e77d46f79a12ddcb864ebfae from peer 892138e3535c4a72b0f0290a6b319d9b ({username='slave'} at 127.0.0.1:45756)
I20240501 08:12:30.375136 28269 tablet_copy_service.cc:161] P e707ad1b8311480f9156e73e14b11059: Beginning new tablet copy session on tablet 577a75e4e77d46f79a12ddcb864ebfae from peer 892138e3535c4a72b0f0290a6b319d9b at {username='slave'} at 127.0.0.1:45756: session id = 892138e3535c4a72b0f0290a6b319d9b-577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:30.377380 28269 tablet_copy_source_session.cc:205] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:30.378523 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577a75e4e77d46f79a12ddcb864ebfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.383877 28888 tablet_copy_client.cc:783] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:30.384171 28888 tablet_copy_client.cc:647] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:30.385601 28892 tablet_copy_client.cc:1002] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Injecting latency into file download: 2000
I20240501 08:12:31.047130 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:31.050246 27943 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:31.050483 27943 raft_consensus.cc:595] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:31.050887 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:31.051090 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:31.450585 28902 raft_consensus.cc:979] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:31.450732 28902 raft_consensus.cc:1067] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:31.451310 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46288
I20240501 08:12:31.451499 28023 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:31.451601 28023 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:31.454214 28023 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:31.454855 28023 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047)
I20240501 08:12:31.455228 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:31.455233 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:31.455435 27871 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:31.455513 27943 raft_consensus.cc:3032] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:12:31.455574 27943 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:31.455792 27943 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:31.456051 27943 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:31.457738 27871 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:31.458106 27988 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:31.458148 27943 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:31.458333 28887 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:12:31.458451 28887 raft_consensus.cc:686] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:31.458639 28887 consensus_queue.cc:237] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:31.461308 27765 catalog_manager.cc:5484] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 3 to 4, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:32.042630 27943 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 08:12:32.043252 28887 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:32.048576 27874 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240501 08:12:32.049156 28910 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:32.233368 28867 tablet_copy_client.cc:523] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:32.236364 28867 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
I20240501 08:12:32.248879 28872 tablet_copy_client.cc:523] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:32.251854 28872 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:32.266264 28877 tablet_copy_client.cc:523] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:32.269646 28877 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap starting.
I20240501 08:12:32.269932 28867 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:32.270203 28867 tablet_bootstrap.cc:492] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:32.270385 28867 ts_tablet_manager.cc:1367] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.034s user 0.035s sys 0.000s
I20240501 08:12:32.270853 28867 raft_consensus.cc:348] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:32.271067 28867 raft_consensus.cc:729] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:32.271226 28867 consensus_queue.cc:260] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:32.271770 28867 ts_tablet_manager.cc:1398] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:12:32.272505 28418 tablet_copy_service.cc:342] P afccd612f3d743a2b54530e51e9a6a4f: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-221cb4dc792649008eb83c64492ef820 received from {username='slave'} at 127.0.0.1:58454
I20240501 08:12:32.272641 28418 tablet_copy_service.cc:433] P afccd612f3d743a2b54530e51e9a6a4f: ending tablet copy session e707ad1b8311480f9156e73e14b11059-221cb4dc792649008eb83c64492ef820 on tablet 221cb4dc792649008eb83c64492ef820 with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:32.289250 28882 tablet_copy_client.cc:523] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:32.291929 28882 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Bootstrap starting.
I20240501 08:12:32.297637 28872 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:32.298099 28872 tablet_bootstrap.cc:492] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:32.298437 28872 ts_tablet_manager.cc:1367] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.047s user 0.047s sys 0.000s
I20240501 08:12:32.299221 28872 raft_consensus.cc:348] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:32.299556 28872 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:32.299788 28872 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:32.301338 28872 ts_tablet_manager.cc:1398] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:32.302457 28034 tablet_copy_service.cc:342] P 2dec92348b7f41c9a5907ddb0f105129: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-40396f8dc9614e369ce570cf51305386 received from {username='slave'} at 127.0.0.1:42712
I20240501 08:12:32.302570 28034 tablet_copy_service.cc:433] P 2dec92348b7f41c9a5907ddb0f105129: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-40396f8dc9614e369ce570cf51305386 on tablet 40396f8dc9614e369ce570cf51305386 with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:32.324860 28877 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:32.325323 28877 tablet_bootstrap.cc:492] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Bootstrap complete.
I20240501 08:12:32.325616 28877 ts_tablet_manager.cc:1367] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Time spent bootstrapping tablet: real 0.056s user 0.029s sys 0.018s
I20240501 08:12:32.326386 28877 raft_consensus.cc:348] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:32.326617 28877 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: afccd612f3d743a2b54530e51e9a6a4f, State: Initialized, Role: LEARNER
I20240501 08:12:32.326826 28877 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: NON_VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: true } }
I20240501 08:12:32.327519 28877 ts_tablet_manager.cc:1398] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:32.328155 27960 tablet_copy_service.cc:342] P 24ffb166dab84eb08b7578c6654deb3c: Request end of tablet copy session afccd612f3d743a2b54530e51e9a6a4f-4e3e31e67e264e9f8f9d07dbc2d90f61 received from {username='slave'} at 127.0.0.1:54262
I20240501 08:12:32.328269 27960 tablet_copy_service.cc:433] P 24ffb166dab84eb08b7578c6654deb3c: ending tablet copy session afccd612f3d743a2b54530e51e9a6a4f-4e3e31e67e264e9f8f9d07dbc2d90f61 on tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 with peer afccd612f3d743a2b54530e51e9a6a4f
I20240501 08:12:32.328336 28258 raft_consensus.cc:1203] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 08:12:32.330982 28882 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:32.331386 28882 tablet_bootstrap.cc:492] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Bootstrap complete.
I20240501 08:12:32.331630 28882 ts_tablet_manager.cc:1367] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.005s
I20240501 08:12:32.332271 28882 raft_consensus.cc:348] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:32.332492 28882 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Initialized, Role: LEARNER
I20240501 08:12:32.332682 28882 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: true } }
I20240501 08:12:32.333284 28882 ts_tablet_manager.cc:1398] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:32.333896 27885 tablet_copy_service.cc:342] P a0210126bd0f40ab9fa38439bb737a0d: Request end of tablet copy session e707ad1b8311480f9156e73e14b11059-947f432452be499595d4472aeb4a6bbe received from {username='slave'} at 127.0.0.1:59638
I20240501 08:12:32.334012 27885 tablet_copy_service.cc:433] P a0210126bd0f40ab9fa38439bb737a0d: ending tablet copy session e707ad1b8311480f9156e73e14b11059-947f432452be499595d4472aeb4a6bbe on tablet 947f432452be499595d4472aeb4a6bbe with peer e707ad1b8311480f9156e73e14b11059
I20240501 08:12:32.364172 28408 raft_consensus.cc:1203] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.26->[3.27-3.27] Dedup: 3.27->[]
I20240501 08:12:32.386364 28888 tablet_copy_client.cc:523] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:32.389413 28888 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap starting.
I20240501 08:12:32.425017 28888 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=93 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:32.425370 28888 tablet_bootstrap.cc:492] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Bootstrap complete.
I20240501 08:12:32.425634 28888 ts_tablet_manager.cc:1367] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Time spent bootstrapping tablet: real 0.036s user 0.033s sys 0.002s
I20240501 08:12:32.426311 28888 raft_consensus.cc:348] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:32.426512 28888 raft_consensus.cc:729] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 892138e3535c4a72b0f0290a6b319d9b, State: Initialized, Role: LEARNER
I20240501 08:12:32.426728 28888 consensus_queue.cc:260] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: NON_VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: true } }
I20240501 08:12:32.427295 28888 ts_tablet_manager.cc:1398] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:32.427922 28269 tablet_copy_service.cc:342] P e707ad1b8311480f9156e73e14b11059: Request end of tablet copy session 892138e3535c4a72b0f0290a6b319d9b-577a75e4e77d46f79a12ddcb864ebfae received from {username='slave'} at 127.0.0.1:45756
I20240501 08:12:32.428015 28269 tablet_copy_service.cc:433] P e707ad1b8311480f9156e73e14b11059: ending tablet copy session 892138e3535c4a72b0f0290a6b319d9b-577a75e4e77d46f79a12ddcb864ebfae on tablet 577a75e4e77d46f79a12ddcb864ebfae with peer 892138e3535c4a72b0f0290a6b319d9b
I20240501 08:12:32.594561 28333 raft_consensus.cc:1203] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 LEARNER]: Deduplicated request from leader. Original: 4.28->[4.29-4.29] Dedup: 4.29->[]
I20240501 08:12:32.840070 28920 raft_consensus.cc:1050] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:32.841212 28920 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:32.843511 28258 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:32.843636 28023 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:32.844148 27871 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:12:32.850477 28568 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:32.850817 28921 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:32.851192 28923 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:32.856077 28923 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.856674 27871 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.861186 28022 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.862339 27765 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.870646 28903 raft_consensus.cc:1050] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: attempting to promote NON_VOTER afccd612f3d743a2b54530e51e9a6a4f to VOTER
I20240501 08:12:32.871105 28258 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.871380 28903 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 25, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:32.874241 28333 raft_consensus.cc:1203] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240501 08:12:32.879350 27871 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:32.880437 28935 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:32.880971 28023 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:32.880971 28407 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 LEARNER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240501 08:12:32.881513 28935 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:32.882320 28935 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:12:32.885818 28023 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.886330 28407 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.885181 28930 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 27 to 28, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.894747 27772 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: config changed from index 27 to 28, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.899008 28259 raft_consensus.cc:1203] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240501 08:12:32.899466 28407 consensus_queue.cc:237] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:32.900184 27871 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } attrs { replace: true } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.904342 27871 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:32.904428 28258 raft_consensus.cc:1261] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Refusing update from remote peer afccd612f3d743a2b54530e51e9a6a4f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:12:32.904937 28921 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:32.905294 28568 consensus_queue.cc:1035] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:32.906910 27943 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:32.908658 28023 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:32.908996 28407 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:12:32.908766 28568 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.909519 28930 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:32.910815 27871 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.911922 27819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 221cb4dc792649008eb83c64492ef820 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:32.912442 28930 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:32.912653 28258 raft_consensus.cc:2932] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:32.913077 27772 catalog_manager.cc:5484] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f reported cstate change: config changed from index 31 to 32, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 2 leader_uuid: "afccd612f3d743a2b54530e51e9a6a4f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:32.915200 28022 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.917845 28408 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.914521 28932 raft_consensus.cc:2932] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.920574 27770 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 28 to 29, VOTER a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) evicted. New cstate: current_term: 3 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } }
I20240501 08:12:32.921525 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cb4dc792649008eb83c64492ef820 with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:32.922251 27759 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:32.925343 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 with delete_type TABLET_DATA_TOMBSTONED (TS a0210126bd0f40ab9fa38439bb737a0d not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:38944
I20240501 08:12:32.925514 28941 tablet_replica.cc:330] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:32.925690 28941 raft_consensus.cc:2229] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.926486 28941 raft_consensus.cc:2258] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.927462 28941 ts_tablet_manager.cc:1875] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:32.929407 28942 tablet_replica.cc:330] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:32.929774 28942 raft_consensus.cc:2229] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.930018 28942 raft_consensus.cc:2258] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.931121 28942 ts_tablet_manager.cc:1875] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:32.933543 28941 ts_tablet_manager.cc:1888] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 08:12:32.933657 28941 log.cc:1196] T 221cb4dc792649008eb83c64492ef820 P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/221cb4dc792649008eb83c64492ef820
I20240501 08:12:32.934465 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet 221cb4dc792649008eb83c64492ef820 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:32.936022 28942 ts_tablet_manager.cc:1888] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20240501 08:12:32.936107 28942 log.cc:1196] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P a0210126bd0f40ab9fa38439bb737a0d: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-0-root/wals/4e3e31e67e264e9f8f9d07dbc2d90f61
I20240501 08:12:32.936699 27758 catalog_manager.cc:4830] TS a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903): tablet 4e3e31e67e264e9f8f9d07dbc2d90f61 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:33.051461 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:33.054735 28022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:33.054966 28022 raft_consensus.cc:595] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:33.055426 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:33.055662 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:33.094146 28943 raft_consensus.cc:1050] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:33.094944 28943 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 27, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:33.097218 27874 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 08:12:33.097432 27943 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 08:12:33.097955 28333 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 LEARNER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 08:12:33.098027 28946 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:33.098392 28943 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:33.098652 28887 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:33.101357 28887 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.102178 27943 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.102404 28333 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.105849 27874 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.106330 27770 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 29 to 30, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.113534 28022 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:33.114917 27874 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:33.115213 28333 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 08:12:33.115703 28943 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:33.116294 28944 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:33.118686 28946 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.119328 28333 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.121165 27871 raft_consensus.cc:2932] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.123392 27757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40396f8dc9614e369ce570cf51305386 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:33.123946 27770 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: config changed from index 30 to 31, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:33.127966 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 40396f8dc9614e369ce570cf51305386 with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:33.128603 28950 tablet_replica.cc:330] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:33.128926 28950 raft_consensus.cc:2229] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.129215 28950 raft_consensus.cc:2258] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.130120 28950 ts_tablet_manager.cc:1875] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:33.134893 28950 ts_tablet_manager.cc:1888] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.30
I20240501 08:12:33.135041 28950 log.cc:1196] T 40396f8dc9614e369ce570cf51305386 P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/40396f8dc9614e369ce570cf51305386
I20240501 08:12:33.135605 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 40396f8dc9614e369ce570cf51305386 (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:33.338023 28951 raft_consensus.cc:1050] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: attempting to promote NON_VOTER e707ad1b8311480f9156e73e14b11059 to VOTER
I20240501 08:12:33.338953 28951 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:33.343187 28259 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 LEARNER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:33.343302 28333 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:33.343822 28934 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:33.343834 28023 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:33.345357 28934 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:33.345883 28934 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:33.348716 28934 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.354511 28023 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.355532 27765 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 28 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:33.354432 28258 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.362438 28333 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e707ad1b8311480f9156e73e14b11059 (127.23.244.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } attrs { replace: true } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.366632 27871 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:33.369220 28332 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:33.370044 28953 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:33.371026 28258 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:12:33.371479 28953 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:33.372779 28901 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.374337 28258 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.375960 27758 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 947f432452be499595d4472aeb4a6bbe with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:33.377007 28332 raft_consensus.cc:2932] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } }
I20240501 08:12:33.377604 27772 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: config changed from index 29 to 30, VOTER 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) evicted. New cstate: current_term: 2 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:33.382370 28011 tablet_service.cc:1514] Processing DeleteTablet for tablet 947f432452be499595d4472aeb4a6bbe with delete_type TABLET_DATA_TOMBSTONED (TS 2dec92348b7f41c9a5907ddb0f105129 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:46270
I20240501 08:12:33.384516 28941 tablet_replica.cc:330] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:33.384698 28941 raft_consensus.cc:2229] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.385198 28941 raft_consensus.cc:2258] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.386010 28941 ts_tablet_manager.cc:1875] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:33.390662 28941 ts_tablet_manager.cc:1888] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240501 08:12:33.390792 28941 log.cc:1196] T 947f432452be499595d4472aeb4a6bbe P 2dec92348b7f41c9a5907ddb0f105129: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-2-root/wals/947f432452be499595d4472aeb4a6bbe
I20240501 08:12:33.391551 27757 catalog_manager.cc:4830] TS 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343): tablet 947f432452be499595d4472aeb4a6bbe (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:33.392889 28944 raft_consensus.cc:979] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:33.393020 28944 raft_consensus.cc:1067] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:33.393553 27943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44018
I20240501 08:12:33.393740 27943 raft_consensus.cc:482] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:33.393834 27943 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:33.395884 27943 raft_consensus.cc:504] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:33.396374 27943 leader_election.cc:288] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 5 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343)
I20240501 08:12:33.397207 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:33.397361 27871 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:33.399505 27871 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 5.
I20240501 08:12:33.399801 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa333a45f3a14263aba9557c7965c605" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:33.399941 28023 raft_consensus.cc:3032] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:12:33.399983 28023 raft_consensus.cc:729] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:33.400234 28023 consensus_queue.cc:260] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:33.400467 28023 raft_consensus.cc:3037] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:33.402650 28023 raft_consensus.cc:2454] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 5.
I20240501 08:12:33.402987 27913 leader_election.cc:302] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:33.403301 28932 raft_consensus.cc:2781] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:12:33.403452 28932 raft_consensus.cc:686] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 5 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:33.403702 28932 consensus_queue.cc:237] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } }
I20240501 08:12:33.406581 27765 catalog_manager.cc:5484] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 4 to 5, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 5 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:33.424842 28958 raft_consensus.cc:1050] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: attempting to promote NON_VOTER 892138e3535c4a72b0f0290a6b319d9b to VOTER
I20240501 08:12:33.425688 28958 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:33.427745 27943 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 08:12:33.427790 28408 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 08:12:33.428195 28332 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEARNER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 08:12:33.428316 28958 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:33.428673 28961 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:33.428975 28959 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:12:33.437074 28958 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.437618 28332 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.437732 28408 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.440392 27943 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.444080 27765 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b reported cstate change: config changed from index 32 to 33, 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } attrs { replace: true } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.449579 28259 consensus_queue.cc:237] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:33.451195 28332 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 08:12:33.451316 28408 raft_consensus.cc:1261] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 08:12:33.451754 28958 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:12:33.452487 28961 consensus_queue.cc:1035] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:12:33.454872 28933 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.455418 28408 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.455499 28332 raft_consensus.cc:2932] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } }
I20240501 08:12:33.457633 27815 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 577a75e4e77d46f79a12ddcb864ebfae with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:33.459093 27770 catalog_manager.cc:5484] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 reported cstate change: config changed from index 33 to 34, VOTER 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) evicted. New cstate: current_term: 2 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:33.462903 27934 tablet_service.cc:1514] Processing DeleteTablet for tablet 577a75e4e77d46f79a12ddcb864ebfae with delete_type TABLET_DATA_TOMBSTONED (TS 24ffb166dab84eb08b7578c6654deb3c not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:43992
I20240501 08:12:33.463227 28950 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:33.463433 28950 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.463776 28950 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.464491 28950 ts_tablet_manager.cc:1875] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:33.468490 28950 ts_tablet_manager.cc:1888] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240501 08:12:33.468575 28950 log.cc:1196] T 577a75e4e77d46f79a12ddcb864ebfae P 24ffb166dab84eb08b7578c6654deb3c: Deleting WAL directory at /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714551036644763-24528-0/minicluster-data/ts-1-root/wals/577a75e4e77d46f79a12ddcb864ebfae
I20240501 08:12:33.469223 27759 catalog_manager.cc:4830] TS 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): tablet 577a75e4e77d46f79a12ddcb864ebfae (table test-workload [id=3fe7a606d7ca488bb588bc8d7b446c5e]) successfully deleted
I20240501 08:12:33.781117 28023 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 08:12:33.781734 28932 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:33.785351 27871 raft_consensus.cc:1261] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240501 08:12:33.786032 28967 consensus_queue.cc:1035] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:35.056074 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:35.058688 27943 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:35.058904 27943 raft_consensus.cc:595] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:35.059491 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:35.059695 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:35.335572 28932 raft_consensus.cc:979] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:35.335691 28932 raft_consensus.cc:1067] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:35.336205 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46288
I20240501 08:12:35.336361 28023 raft_consensus.cc:482] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:35.336424 28023 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:35.338248 28023 raft_consensus.cc:504] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:35.338819 28023 leader_election.cc:288] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837)
I20240501 08:12:35.339223 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:35.339358 27943 raft_consensus.cc:3032] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:12:35.339437 27943 raft_consensus.cc:729] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:35.339352 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3e31e67e264e9f8f9d07dbc2d90f61" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
I20240501 08:12:35.339589 28408 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:35.339659 27943 consensus_queue.cc:260] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:35.339977 27943 raft_consensus.cc:3037] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:12:35.341442 28408 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:35.341640 27943 raft_consensus.cc:2454] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 4.
I20240501 08:12:35.341843 27989 leader_election.cc:302] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, afccd612f3d743a2b54530e51e9a6a4f; no voters:
I20240501 08:12:35.342089 28944 raft_consensus.cc:2781] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:12:35.342418 28944 raft_consensus.cc:686] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:35.342618 28944 consensus_queue.cc:237] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:35.344678 27764 catalog_manager.cc:5484] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 3 to 4, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 4 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.723587 27943 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 08:12:35.724097 28944 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:35.728154 28408 raft_consensus.cc:1261] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 08:12:35.728703 28973 consensus_queue.cc:1035] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:37.060050 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:37.062919 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:37.063150 28023 raft_consensus.cc:595] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Received request to transfer leadership to TS 24ffb166dab84eb08b7578c6654deb3c
I20240501 08:12:37.063673 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:37.063884 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:37.373961 28984 raft_consensus.cc:979] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:37.374143 28984 raft_consensus.cc:1067] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Signalling peer 24ffb166dab84eb08b7578c6654deb3c to start an election
I20240501 08:12:37.374725 27943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
from {username='slave'} at 127.0.0.1:44018
I20240501 08:12:37.374948 27943 raft_consensus.cc:482] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:37.375073 27943 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:37.376978 27943 raft_consensus.cc:504] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:37.378036 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:37.378098 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "24ffb166dab84eb08b7578c6654deb3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
I20240501 08:12:37.378199 28023 raft_consensus.cc:3032] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:37.377861 27943 leader_election.cc:288] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 2 election: Requested vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837)
I20240501 08:12:37.378350 28023 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:37.378676 28023 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:37.378971 28023 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:37.380878 28023 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 2.
I20240501 08:12:37.378350 28408 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:37.381321 27913 leader_election.cc:302] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ffb166dab84eb08b7578c6654deb3c, 2dec92348b7f41c9a5907ddb0f105129; no voters:
I20240501 08:12:37.381584 28932 raft_consensus.cc:2781] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:37.381898 28932 raft_consensus.cc:686] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Becoming Leader. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:37.382109 28932 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:37.383199 28408 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ffb166dab84eb08b7578c6654deb3c in term 2.
I20240501 08:12:37.384353 27764 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c reported cstate change: term changed from 1 to 2, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2). New cstate: current_term: 2 leader_uuid: "24ffb166dab84eb08b7578c6654deb3c" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.768144 28023 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:37.768748 28932 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:37.773149 28408 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Refusing update from remote peer 24ffb166dab84eb08b7578c6654deb3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:12:37.773672 28932 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:12:39.064302 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:39.067142 27943 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69"
dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
mode: GRACEFUL
new_leader_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:42204
I20240501 08:12:39.067373 27943 raft_consensus.cc:595] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Received request to transfer leadership to TS 2dec92348b7f41c9a5907ddb0f105129
I20240501 08:12:39.067814 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:39.067998 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:39.253988 29001 raft_consensus.cc:979] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: : Instructing follower 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:39.254182 28932 raft_consensus.cc:1067] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Signalling peer 2dec92348b7f41c9a5907ddb0f105129 to start an election
I20240501 08:12:39.254746 28023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
from {username='slave'} at 127.0.0.1:46288
I20240501 08:12:39.254915 28023 raft_consensus.cc:482] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:39.255036 28023 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:39.256975 28023 raft_consensus.cc:504] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:39.257516 28023 leader_election.cc:288] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 3 election: Requested vote from peers 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047), afccd612f3d743a2b54530e51e9a6a4f (127.23.244.6:45837)
I20240501 08:12:39.257889 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "24ffb166dab84eb08b7578c6654deb3c"
I20240501 08:12:39.257982 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25deaffcb0e441e590d7b300e1dcda69" candidate_uuid: "2dec92348b7f41c9a5907ddb0f105129" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "afccd612f3d743a2b54530e51e9a6a4f"
I20240501 08:12:39.258090 27943 raft_consensus.cc:3032] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:39.258183 28408 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:39.258211 27943 raft_consensus.cc:729] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24ffb166dab84eb08b7578c6654deb3c, State: Running, Role: LEADER
I20240501 08:12:39.258452 27943 consensus_queue.cc:260] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:39.258759 27943 raft_consensus.cc:3037] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:39.260031 28408 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 3.
I20240501 08:12:39.260406 27989 leader_election.cc:302] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dec92348b7f41c9a5907ddb0f105129, afccd612f3d743a2b54530e51e9a6a4f; no voters:
I20240501 08:12:39.260526 27943 raft_consensus.cc:2454] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dec92348b7f41c9a5907ddb0f105129 in term 3.
I20240501 08:12:39.260883 28984 raft_consensus.cc:2781] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:39.261371 28984 raft_consensus.cc:686] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 LEADER]: Becoming Leader. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:39.261646 28984 consensus_queue.cc:237] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } }
I20240501 08:12:39.264153 27764 catalog_manager.cc:5484] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 reported cstate change: term changed from 2 to 3, leader changed from 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2) to 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3). New cstate: current_term: 3 leader_uuid: "2dec92348b7f41c9a5907ddb0f105129" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:39.695232 28408 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:39.695761 28984 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afccd612f3d743a2b54530e51e9a6a4f" member_type: VOTER last_known_addr { host: "127.23.244.6" port: 45837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:39.701717 27943 raft_consensus.cc:1261] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Refusing update from remote peer 2dec92348b7f41c9a5907ddb0f105129: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:12:39.702399 29011 consensus_queue.cc:1035] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ffb166dab84eb08b7578c6654deb3c" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:12:41.068428 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:41.071452 28023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
mode: GRACEFUL
new_leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:46294
I20240501 08:12:41.071667 28023 raft_consensus.cc:595] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Received request to transfer leadership to TS a0210126bd0f40ab9fa38439bb737a0d
I20240501 08:12:41.072149 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:41.072369 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:41.439990 29002 raft_consensus.cc:979] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: : Instructing follower a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:41.440176 29002 raft_consensus.cc:1067] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Signalling peer a0210126bd0f40ab9fa38439bb737a0d to start an election
I20240501 08:12:41.440704 27874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40396f8dc9614e369ce570cf51305386"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
from {username='slave'} at 127.0.0.1:38970
I20240501 08:12:41.440928 27874 raft_consensus.cc:482] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:41.441044 27874 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:41.442902 27874 raft_consensus.cc:504] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:41.443459 27874 leader_election.cc:288] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 5 election: Requested vote from peers 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3:36343), 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:41.443831 28023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "2dec92348b7f41c9a5907ddb0f105129"
I20240501 08:12:41.443928 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40396f8dc9614e369ce570cf51305386" candidate_uuid: "a0210126bd0f40ab9fa38439bb737a0d" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
I20240501 08:12:41.444025 28023 raft_consensus.cc:3032] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:12:41.444075 28332 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:41.444121 28023 raft_consensus.cc:729] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2dec92348b7f41c9a5907ddb0f105129, State: Running, Role: LEADER
I20240501 08:12:41.444375 28023 consensus_queue.cc:260] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:41.444756 28023 raft_consensus.cc:3037] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:12:41.446843 28332 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 5.
I20240501 08:12:41.447266 27840 leader_election.cc:302] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892138e3535c4a72b0f0290a6b319d9b, a0210126bd0f40ab9fa38439bb737a0d; no voters:
I20240501 08:12:41.447382 28023 raft_consensus.cc:2454] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0210126bd0f40ab9fa38439bb737a0d in term 5.
I20240501 08:12:41.447577 29000 raft_consensus.cc:2781] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:12:41.447975 29000 raft_consensus.cc:686] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 5 LEADER]: Becoming Leader. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:41.448246 29000 consensus_queue.cc:237] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } }
I20240501 08:12:41.450556 27772 catalog_manager.cc:5484] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d reported cstate change: term changed from 4 to 5, leader changed from 2dec92348b7f41c9a5907ddb0f105129 (127.23.244.3) to a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1). New cstate: current_term: 5 leader_uuid: "a0210126bd0f40ab9fa38439bb737a0d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.924168 28023 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240501 08:12:41.924751 29000 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dec92348b7f41c9a5907ddb0f105129" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 36343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:41.928452 28332 raft_consensus.cc:1261] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Refusing update from remote peer a0210126bd0f40ab9fa38439bb737a0d: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240501 08:12:41.929109 29025 consensus_queue.cc:1035] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:12:43.072939 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:43.076041 27871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe"
dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
mode: GRACEFUL
new_leader_uuid: "e707ad1b8311480f9156e73e14b11059"
from {username='slave'} at 127.0.0.1:59674
I20240501 08:12:43.076278 27871 raft_consensus.cc:595] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Received request to transfer leadership to TS e707ad1b8311480f9156e73e14b11059
I20240501 08:12:43.076784 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:12:43.077013 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:43.524471 29031 raft_consensus.cc:979] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: : Instructing follower e707ad1b8311480f9156e73e14b11059 to start an election
I20240501 08:12:43.524874 29025 raft_consensus.cc:1067] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Signalling peer e707ad1b8311480f9156e73e14b11059 to start an election
I20240501 08:12:43.525492 28258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe"
dest_uuid: "e707ad1b8311480f9156e73e14b11059"
from {username='slave'} at 127.0.0.1:45742
I20240501 08:12:43.525684 28258 raft_consensus.cc:482] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:43.525794 28258 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:43.527719 28258 raft_consensus.cc:504] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:43.528345 28258 leader_election.cc:288] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [CANDIDATE]: Term 3 election: Requested vote from peers a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1:37903), 892138e3535c4a72b0f0290a6b319d9b (127.23.244.5:38381)
I20240501 08:12:43.528703 27871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "e707ad1b8311480f9156e73e14b11059" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "a0210126bd0f40ab9fa38439bb737a0d"
I20240501 08:12:43.528741 28332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947f432452be499595d4472aeb4a6bbe" candidate_uuid: "e707ad1b8311480f9156e73e14b11059" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "892138e3535c4a72b0f0290a6b319d9b"
I20240501 08:12:43.528955 27871 raft_consensus.cc:3032] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:43.528960 28332 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:43.529069 27871 raft_consensus.cc:729] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0210126bd0f40ab9fa38439bb737a0d, State: Running, Role: LEADER
I20240501 08:12:43.529310 27871 consensus_queue.cc:260] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:43.529614 27871 raft_consensus.cc:3037] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:43.530895 28332 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e707ad1b8311480f9156e73e14b11059 in term 3.
I20240501 08:12:43.531260 28224 leader_election.cc:302] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892138e3535c4a72b0f0290a6b319d9b, e707ad1b8311480f9156e73e14b11059; no voters:
I20240501 08:12:43.531514 29030 raft_consensus.cc:2781] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:12:43.531623 27871 raft_consensus.cc:2454] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e707ad1b8311480f9156e73e14b11059 in term 3.
I20240501 08:12:43.532074 29030 raft_consensus.cc:686] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 3 LEADER]: Becoming Leader. State: Replica: e707ad1b8311480f9156e73e14b11059, State: Running, Role: LEADER
I20240501 08:12:43.532322 29030 consensus_queue.cc:237] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } }
I20240501 08:12:43.534605 27764 catalog_manager.cc:5484] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 reported cstate change: term changed from 2 to 3, leader changed from a0210126bd0f40ab9fa38439bb737a0d (127.23.244.1) to e707ad1b8311480f9156e73e14b11059 (127.23.244.4). New cstate: current_term: 3 leader_uuid: "e707ad1b8311480f9156e73e14b11059" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e707ad1b8311480f9156e73e14b11059" member_type: VOTER last_known_addr { host: "127.23.244.4" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:44.018957 27871 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:12:44.019575 29030 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0210126bd0f40ab9fa38439bb737a0d" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:12:44.023741 28332 raft_consensus.cc:1261] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Refusing update from remote peer e707ad1b8311480f9156e73e14b11059: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:12:44.025600 29042 consensus_queue.cc:1035] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892138e3535c4a72b0f0290a6b319d9b" member_type: VOTER last_known_addr { host: "127.23.244.5" port: 38381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20240501 08:12:45.077430 27826 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:45.079391 27826 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:45.079593 27826 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:45.478639 24528 tablet_server.cc:178] TabletServer@127.23.244.1:0 shutting down...
I20240501 08:12:45.506661 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.506911 24528 tablet_replica.cc:330] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.507109 24528 raft_consensus.cc:2229] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 5 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.507345 24528 raft_consensus.cc:2258] T 40396f8dc9614e369ce570cf51305386 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.507863 24528 tablet_replica.cc:330] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.507972 24528 raft_consensus.cc:2229] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.508152 24528 raft_consensus.cc:2258] T 221cb4dc792649008eb83c64492ef820 P a0210126bd0f40ab9fa38439bb737a0d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.508538 24528 tablet_replica.cc:330] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.508641 24528 raft_consensus.cc:2229] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.508877 24528 raft_consensus.cc:2258] T 9a506a24817641c8afbfe3f8a5e696c0 P a0210126bd0f40ab9fa38439bb737a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.509263 24528 tablet_replica.cc:330] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.509353 24528 raft_consensus.cc:2229] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.509452 24528 raft_consensus.cc:2258] T aa333a45f3a14263aba9557c7965c605 P a0210126bd0f40ab9fa38439bb737a0d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.509802 24528 tablet_replica.cc:330] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.509909 24528 raft_consensus.cc:2229] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.510147 24528 raft_consensus.cc:2258] T df29fad2de5b4ac18e52d233ee63a6d0 P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.510565 24528 tablet_replica.cc:330] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d: stopping tablet replica
I20240501 08:12:45.510699 24528 raft_consensus.cc:2229] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.510813 24528 raft_consensus.cc:2258] T 947f432452be499595d4472aeb4a6bbe P a0210126bd0f40ab9fa38439bb737a0d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.527915 24528 tablet_server.cc:194] TabletServer@127.23.244.1:0 shutdown complete.
I20240501 08:12:45.533166 24528 tablet_server.cc:178] TabletServer@127.23.244.2:0 shutting down...
I20240501 08:12:45.541195 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.541440 24528 tablet_replica.cc:330] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.541611 24528 raft_consensus.cc:2229] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.541776 24528 raft_consensus.cc:2258] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.542358 24528 tablet_replica.cc:330] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.542515 24528 raft_consensus.cc:2229] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.542649 24528 raft_consensus.cc:2258] T 21c5ff99f3f94039a7ba50580e56f0ff P 24ffb166dab84eb08b7578c6654deb3c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.543099 24528 tablet_replica.cc:330] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.543228 24528 raft_consensus.cc:2229] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.543470 24528 raft_consensus.cc:2258] T df29fad2de5b4ac18e52d233ee63a6d0 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.543897 24528 tablet_replica.cc:330] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.544018 24528 raft_consensus.cc:2229] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.544245 24528 raft_consensus.cc:2258] T 7af4b37f38284e8d9cdaaa56247b96e3 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.544646 24528 tablet_replica.cc:330] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.544790 24528 raft_consensus.cc:2229] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 5 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.545017 24528 raft_consensus.cc:2258] T aa333a45f3a14263aba9557c7965c605 P 24ffb166dab84eb08b7578c6654deb3c [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.545430 24528 tablet_replica.cc:330] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c: stopping tablet replica
I20240501 08:12:45.545549 24528 raft_consensus.cc:2229] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.545665 24528 raft_consensus.cc:2258] T 25deaffcb0e441e590d7b300e1dcda69 P 24ffb166dab84eb08b7578c6654deb3c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.576687 24528 tablet_server.cc:194] TabletServer@127.23.244.2:0 shutdown complete.
I20240501 08:12:45.582105 24528 tablet_server.cc:178] TabletServer@127.23.244.3:0 shutting down...
I20240501 08:12:45.592075 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.592329 24528 tablet_replica.cc:330] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:45.592510 24528 raft_consensus.cc:2229] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.592792 24528 raft_consensus.cc:2258] T 9121766525b940d58127de0d7b9f5d57 P 2dec92348b7f41c9a5907ddb0f105129 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.593348 24528 tablet_replica.cc:330] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:45.593482 24528 raft_consensus.cc:2229] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.593765 24528 raft_consensus.cc:2258] T 25deaffcb0e441e590d7b300e1dcda69 P 2dec92348b7f41c9a5907ddb0f105129 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.594246 24528 tablet_replica.cc:330] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:45.594377 24528 raft_consensus.cc:2229] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.594590 24528 raft_consensus.cc:2258] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:45.594676 28373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.244.2:46047: connect: Connection refused (error 111)
I20240501 08:12:45.595041 24528 tablet_replica.cc:330] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:45.595172 24528 raft_consensus.cc:2229] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.595299 24528 raft_consensus.cc:2258] T aa333a45f3a14263aba9557c7965c605 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.595714 24528 tablet_replica.cc:330] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
W20240501 08:12:45.595806 28373 consensus_peers.cc:485] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f -> Peer 24ffb166dab84eb08b7578c6654deb3c (127.23.244.2:46047): Couldn't send request to peer 24ffb166dab84eb08b7578c6654deb3c. Status: Network error: Client connection negotiation failed: client connection to 127.23.244.2:46047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.595835 24528 raft_consensus.cc:2229] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.595958 24528 raft_consensus.cc:2258] T 21c5ff99f3f94039a7ba50580e56f0ff P 2dec92348b7f41c9a5907ddb0f105129 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.596367 24528 tablet_replica.cc:330] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129: stopping tablet replica
I20240501 08:12:45.596462 24528 raft_consensus.cc:2229] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.596570 24528 raft_consensus.cc:2258] T 40396f8dc9614e369ce570cf51305386 P 2dec92348b7f41c9a5907ddb0f105129 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.612450 24528 tablet_server.cc:194] TabletServer@127.23.244.3:0 shutdown complete.
I20240501 08:12:45.617874 24528 tablet_server.cc:178] TabletServer@127.23.244.4:0 shutting down...
I20240501 08:12:45.642478 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.642822 24528 tablet_replica.cc:330] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.643020 24528 raft_consensus.cc:2229] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.643277 24528 raft_consensus.cc:2258] T 221cb4dc792649008eb83c64492ef820 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.643941 24528 tablet_replica.cc:330] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.644106 24528 raft_consensus.cc:2229] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.644305 24528 raft_consensus.cc:2258] T 9121766525b940d58127de0d7b9f5d57 P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.644840 24528 tablet_replica.cc:330] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.644979 24528 raft_consensus.cc:2229] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 3 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.645259 24528 raft_consensus.cc:2258] T 947f432452be499595d4472aeb4a6bbe P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.645718 24528 tablet_replica.cc:330] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.645866 24528 raft_consensus.cc:2229] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.646042 24528 raft_consensus.cc:2258] T df29fad2de5b4ac18e52d233ee63a6d0 P e707ad1b8311480f9156e73e14b11059 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.646499 24528 tablet_replica.cc:330] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.646631 24528 raft_consensus.cc:2229] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.646811 24528 raft_consensus.cc:2258] T 9a506a24817641c8afbfe3f8a5e696c0 P e707ad1b8311480f9156e73e14b11059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.647305 24528 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059: stopping tablet replica
I20240501 08:12:45.647415 24528 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.647665 24528 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P e707ad1b8311480f9156e73e14b11059 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.663010 24528 tablet_server.cc:194] TabletServer@127.23.244.4:0 shutdown complete.
I20240501 08:12:45.667125 24528 tablet_server.cc:178] TabletServer@127.23.244.5:0 shutting down...
I20240501 08:12:45.675010 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.675257 24528 tablet_replica.cc:330] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.675447 24528 raft_consensus.cc:2229] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.675590 24528 raft_consensus.cc:2258] T 947f432452be499595d4472aeb4a6bbe P 892138e3535c4a72b0f0290a6b319d9b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.676116 24528 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.676230 24528 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.676401 24528 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.676841 24528 tablet_replica.cc:330] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.676954 24528 raft_consensus.cc:2229] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.677045 24528 raft_consensus.cc:2258] T 40396f8dc9614e369ce570cf51305386 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.677647 24528 tablet_replica.cc:330] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.677769 24528 raft_consensus.cc:2229] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.678006 24528 raft_consensus.cc:2258] T 9a506a24817641c8afbfe3f8a5e696c0 P 892138e3535c4a72b0f0290a6b319d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.678510 24528 tablet_replica.cc:330] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.678620 24528 raft_consensus.cc:2229] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.678889 24528 raft_consensus.cc:2258] T 7af4b37f38284e8d9cdaaa56247b96e3 P 892138e3535c4a72b0f0290a6b319d9b [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.679314 24528 tablet_replica.cc:330] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b: stopping tablet replica
I20240501 08:12:45.679422 24528 raft_consensus.cc:2229] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.679631 24528 raft_consensus.cc:2258] T 9121766525b940d58127de0d7b9f5d57 P 892138e3535c4a72b0f0290a6b319d9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.694252 24528 tablet_server.cc:194] TabletServer@127.23.244.5:0 shutdown complete.
I20240501 08:12:45.698194 24528 tablet_server.cc:178] TabletServer@127.23.244.6:0 shutting down...
I20240501 08:12:45.706640 24528 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:12:45.706925 24528 tablet_replica.cc:330] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.707098 24528 raft_consensus.cc:2229] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.707393 24528 raft_consensus.cc:2258] T 577a75e4e77d46f79a12ddcb864ebfae P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.707957 24528 tablet_replica.cc:330] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.708093 24528 raft_consensus.cc:2229] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.708266 24528 raft_consensus.cc:2258] T 7af4b37f38284e8d9cdaaa56247b96e3 P afccd612f3d743a2b54530e51e9a6a4f [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.708683 24528 tablet_replica.cc:330] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.708834 24528 raft_consensus.cc:2229] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 4 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.709076 24528 raft_consensus.cc:2258] T 21c5ff99f3f94039a7ba50580e56f0ff P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.709497 24528 tablet_replica.cc:330] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.709612 24528 raft_consensus.cc:2229] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.709719 24528 raft_consensus.cc:2258] T 4e3e31e67e264e9f8f9d07dbc2d90f61 P afccd612f3d743a2b54530e51e9a6a4f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.710093 24528 tablet_replica.cc:330] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.710223 24528 raft_consensus.cc:2229] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.710343 24528 raft_consensus.cc:2258] T 25deaffcb0e441e590d7b300e1dcda69 P afccd612f3d743a2b54530e51e9a6a4f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.710714 24528 tablet_replica.cc:330] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f: stopping tablet replica
I20240501 08:12:45.710832 24528 raft_consensus.cc:2229] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.711053 24528 raft_consensus.cc:2258] T 221cb4dc792649008eb83c64492ef820 P afccd612f3d743a2b54530e51e9a6a4f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.725410 24528 tablet_server.cc:194] TabletServer@127.23.244.6:0 shutdown complete.
I20240501 08:12:45.729744 24528 master.cc:537] Master@127.23.244.62:38837 shutting down...
I20240501 08:12:45.751220 24528 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:45.751464 24528 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.751562 24528 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7cfb0dd38cc6424a914275b0b8b4fb1d: stopping tablet replica
I20240501 08:12:45.764756 24528 master.cc:552] Master@127.23.244.62:38837 shutdown complete.
[ OK ] AutoRebalancerTest.TestMaxMovesPerServer (35285 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240501 08:12:45.787400 24528 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.244.62:43427
I20240501 08:12:45.787856 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:45.790951 29048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.791386 29047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.791973 29050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:45.792688 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:45.792922 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:45.792970 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:45.792996 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551165792993 us; error 0 us; skew 500 ppm
I20240501 08:12:45.793164 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:45.794353 24528 webserver.cc:460] Webserver started at http://127.23.244.62:45825/ using document root <none> and password file <none>
I20240501 08:12:45.794554 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:45.794615 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:45.794741 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:45.795359 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/master-0-root/instance:
uuid: "bd11e41843dd48c0ae96dff4f7440eeb"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.798171 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:45.799647 29055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.799985 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:12:45.800066 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/master-0-root
uuid: "bd11e41843dd48c0ae96dff4f7440eeb"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.800156 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:45.813198 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:45.813665 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:45.823805 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.62:43427
I20240501 08:12:45.823820 29106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.62:43427 every 8 connection(s)
I20240501 08:12:45.825001 29107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.828927 29107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb: Bootstrap starting.
I20240501 08:12:45.830065 29107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:45.831456 29107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb: No bootstrap required, opened a new log
I20240501 08:12:45.831936 29107 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER }
I20240501 08:12:45.832067 29107 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:45.832116 29107 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd11e41843dd48c0ae96dff4f7440eeb, State: Initialized, Role: FOLLOWER
I20240501 08:12:45.832283 29107 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER }
I20240501 08:12:45.832402 29107 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:12:45.832442 29107 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:12:45.832489 29107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:45.834142 29107 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER }
I20240501 08:12:45.834296 29107 leader_election.cc:302] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd11e41843dd48c0ae96dff4f7440eeb; no voters:
I20240501 08:12:45.834599 29107 leader_election.cc:288] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:12:45.834712 29110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:45.835093 29110 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [term 1 LEADER]: Becoming Leader. State: Replica: bd11e41843dd48c0ae96dff4f7440eeb, State: Running, Role: LEADER
I20240501 08:12:45.835338 29110 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER }
I20240501 08:12:45.835425 29107 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:12:45.836000 29111 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER } }
I20240501 08:12:45.836068 29112 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd11e41843dd48c0ae96dff4f7440eeb. Latest consensus state: current_term: 1 leader_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11e41843dd48c0ae96dff4f7440eeb" member_type: VOTER } }
I20240501 08:12:45.836258 29111 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:45.836372 29112 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:45.837370 29115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:12:45.838575 29115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:12:45.840190 24528 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:12:45.841261 29115 catalog_manager.cc:1347] Generated new cluster ID: 96dd8010572842f3b3513347fc5e5805
I20240501 08:12:45.841334 29115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:12:45.854600 29115 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:12:45.855316 29115 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:12:45.862174 29115 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd11e41843dd48c0ae96dff4f7440eeb: Generated new TSK 0
I20240501 08:12:45.862365 29115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:12:45.872830 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:45.875476 29128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.876041 29129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.876546 29131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:45.877435 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:45.877661 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:45.877715 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:45.877746 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551165877742 us; error 0 us; skew 500 ppm
I20240501 08:12:45.877872 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:45.879122 24528 webserver.cc:460] Webserver started at http://127.23.244.1:34939/ using document root <none> and password file <none>
I20240501 08:12:45.879319 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:45.879374 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:45.879464 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:45.880062 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-0-root/instance:
uuid: "4bea04874bc846c1ac55ded6f9df08a5"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.882498 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:45.884024 29136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.884351 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:45.884455 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-0-root
uuid: "4bea04874bc846c1ac55ded6f9df08a5"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.884562 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:45.896005 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:45.896495 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:45.897049 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:45.897862 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:45.897914 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.897987 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:45.898025 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.911324 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.1:41213
I20240501 08:12:45.911351 29198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.1:41213 every 8 connection(s)
I20240501 08:12:45.912933 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:45.915825 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.916519 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.917104 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:45.917487 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:45.917709 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:45.917760 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:45.917783 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551165917779 us; error 0 us; skew 500 ppm
I20240501 08:12:45.917903 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:45.919154 29199 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43427
I20240501 08:12:45.919291 29199 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:45.919605 29199 heartbeater.cc:511] Master 127.23.244.62:43427 requested a full tablet report, sending...
I20240501 08:12:45.919996 24528 webserver.cc:460] Webserver started at http://127.23.244.2:42339/ using document root <none> and password file <none>
I20240501 08:12:45.920243 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:45.920320 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:45.920430 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:45.920521 29072 ts_manager.cc:194] Registered new tserver with Master: 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213)
I20240501 08:12:45.921257 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-1-root/instance:
uuid: "ae8bea7295884bc8bdc0d1c689469255"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.921986 29072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39090
I20240501 08:12:45.923699 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:45.925072 29211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.925374 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:45.925467 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-1-root
uuid: "ae8bea7295884bc8bdc0d1c689469255"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.925558 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:45.939249 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:45.939780 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:45.940310 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:45.941219 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:45.941282 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.941365 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:45.941404 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.954457 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.2:46603
I20240501 08:12:45.954491 29273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.2:46603 every 8 connection(s)
I20240501 08:12:45.955816 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:45.959085 29278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.959513 29277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:45.959851 29280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:45.960616 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:45.960886 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:45.960955 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:45.960987 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551165960983 us; error 0 us; skew 500 ppm
I20240501 08:12:45.961166 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:45.961985 29274 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43427
I20240501 08:12:45.962102 29274 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:45.962390 29274 heartbeater.cc:511] Master 127.23.244.62:43427 requested a full tablet report, sending...
I20240501 08:12:45.962531 24528 webserver.cc:460] Webserver started at http://127.23.244.3:34617/ using document root <none> and password file <none>
I20240501 08:12:45.962766 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:45.962862 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:45.962977 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:45.963297 29072 ts_manager.cc:194] Registered new tserver with Master: ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603)
I20240501 08:12:45.963600 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-2-root/instance:
uuid: "86bc30284a7e44aa9f9b18e530a7d7db"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.964035 29072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39104
I20240501 08:12:45.966092 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20240501 08:12:45.967538 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.967809 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:45.967922 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-2-root
uuid: "86bc30284a7e44aa9f9b18e530a7d7db"
format_stamp: "Formatted at 2024-05-01 08:12:45 on dist-test-slave-fck4"
I20240501 08:12:45.968010 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:45.982131 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:45.982656 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:45.983153 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:45.983922 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:45.983974 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.984040 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:45.984071 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:45.995536 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.3:39221
I20240501 08:12:45.995558 29347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.3:39221 every 8 connection(s)
I20240501 08:12:46.001631 29348 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43427
I20240501 08:12:46.001760 29348 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:46.002051 29348 heartbeater.cc:511] Master 127.23.244.62:43427 requested a full tablet report, sending...
I20240501 08:12:46.002784 29072 ts_manager.cc:194] Registered new tserver with Master: 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:46.003305 24528 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006812546s
I20240501 08:12:46.003461 29072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39114
I20240501 08:12:46.923123 29199 heartbeater.cc:503] Master 127.23.244.62:43427 was elected leader, sending a full tablet report...
I20240501 08:12:46.965260 29274 heartbeater.cc:503] Master 127.23.244.62:43427 was elected leader, sending a full tablet report...
I20240501 08:12:47.004710 29348 heartbeater.cc:503] Master 127.23.244.62:43427 was elected leader, sending a full tablet report...
I20240501 08:12:47.030933 24528 test_util.cc:268] Using random seed: 222467728
I20240501 08:12:47.043597 29072 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39126:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:12:47.044226 29072 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:12:47.090869 29164 tablet_service.cc:1467] Processing CreateTablet for tablet af7b8f097fe54bdab219a389e622c587 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:12:47.091120 29313 tablet_service.cc:1467] Processing CreateTablet for tablet af7b8f097fe54bdab219a389e622c587 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:12:47.091375 29164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7b8f097fe54bdab219a389e622c587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.091442 29313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7b8f097fe54bdab219a389e622c587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.091496 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 7495cdffb96945acba22f8450c9917ff (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:12:47.091478 29239 tablet_service.cc:1467] Processing CreateTablet for tablet af7b8f097fe54bdab219a389e622c587 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 08:12:47.091948 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7495cdffb96945acba22f8450c9917ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.091948 29239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7b8f097fe54bdab219a389e622c587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.097046 29312 tablet_service.cc:1467] Processing CreateTablet for tablet 7495cdffb96945acba22f8450c9917ff (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:12:47.097395 29312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7495cdffb96945acba22f8450c9917ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.098418 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 944204304b9a46b4a46788a5f6dd23a6 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:12:47.098762 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944204304b9a46b4a46788a5f6dd23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.101157 29311 tablet_service.cc:1467] Processing CreateTablet for tablet 944204304b9a46b4a46788a5f6dd23a6 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:12:47.101676 29311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944204304b9a46b4a46788a5f6dd23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.104910 29368 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5: Bootstrap starting.
I20240501 08:12:47.107236 29368 tablet_bootstrap.cc:654] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.107882 29370 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db: Bootstrap starting.
I20240501 08:12:47.108383 29369 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255: Bootstrap starting.
I20240501 08:12:47.109635 29370 tablet_bootstrap.cc:654] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.109926 29239 tablet_service.cc:1467] Processing CreateTablet for tablet 7495cdffb96945acba22f8450c9917ff (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 08:12:47.110028 29238 tablet_service.cc:1467] Processing CreateTablet for tablet 944204304b9a46b4a46788a5f6dd23a6 (DEFAULT_TABLE table=test-workload [id=e8054cc0559c4ec2a189612e0b79b34d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 08:12:47.110440 29239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7495cdffb96945acba22f8450c9917ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.110833 29369 tablet_bootstrap.cc:654] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.110894 29238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944204304b9a46b4a46788a5f6dd23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.112622 29368 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5: No bootstrap required, opened a new log
I20240501 08:12:47.112809 29368 ts_tablet_manager.cc:1367] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:12:47.113556 29368 raft_consensus.cc:348] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.113834 29368 raft_consensus.cc:374] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.113934 29368 raft_consensus.cc:729] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea04874bc846c1ac55ded6f9df08a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.114207 29368 consensus_queue.cc:260] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.120056 29368 ts_tablet_manager.cc:1398] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:47.120126 29369 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255: No bootstrap required, opened a new log
I20240501 08:12:47.120354 29368 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: Bootstrap starting.
I20240501 08:12:47.120383 29369 ts_tablet_manager.cc:1367] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 08:12:47.121486 29369 raft_consensus.cc:348] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.121814 29369 raft_consensus.cc:374] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.121951 29369 raft_consensus.cc:729] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8bea7295884bc8bdc0d1c689469255, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.122385 29368 tablet_bootstrap.cc:654] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.122339 29369 consensus_queue.cc:260] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.124219 29370 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db: No bootstrap required, opened a new log
I20240501 08:12:47.124341 29370 ts_tablet_manager.cc:1367] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240501 08:12:47.124475 29368 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: No bootstrap required, opened a new log
I20240501 08:12:47.124630 29368 ts_tablet_manager.cc:1367] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:47.124946 29370 raft_consensus.cc:348] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.125082 29370 raft_consensus.cc:374] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.125181 29370 raft_consensus.cc:729] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc30284a7e44aa9f9b18e530a7d7db, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.125386 29370 consensus_queue.cc:260] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.125465 29368 raft_consensus.cc:348] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.125746 29368 raft_consensus.cc:374] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.125825 29368 raft_consensus.cc:729] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea04874bc846c1ac55ded6f9df08a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.126055 29368 consensus_queue.cc:260] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.126665 29368 ts_tablet_manager.cc:1398] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:47.127038 29368 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5: Bootstrap starting.
I20240501 08:12:47.129057 29368 tablet_bootstrap.cc:654] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.129516 29369 ts_tablet_manager.cc:1398] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240501 08:12:47.130604 29369 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255: Bootstrap starting.
I20240501 08:12:47.132046 29374 raft_consensus.cc:482] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:47.132172 29374 raft_consensus.cc:504] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.132936 29374 leader_election.cc:288] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603), 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:47.133131 29370 ts_tablet_manager.cc:1398] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:12:47.133375 29370 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db: Bootstrap starting.
I20240501 08:12:47.134891 29370 tablet_bootstrap.cc:654] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.138159 29369 tablet_bootstrap.cc:654] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.140146 29369 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255: No bootstrap required, opened a new log
I20240501 08:12:47.140303 29369 ts_tablet_manager.cc:1367] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 08:12:47.141008 29369 raft_consensus.cc:348] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.141201 29369 raft_consensus.cc:374] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.141256 29369 raft_consensus.cc:729] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8bea7295884bc8bdc0d1c689469255, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.141471 29369 consensus_queue.cc:260] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.141980 29369 ts_tablet_manager.cc:1398] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:47.146014 29369 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255: Bootstrap starting.
I20240501 08:12:47.147938 29369 tablet_bootstrap.cc:654] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.149566 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bea7295884bc8bdc0d1c689469255" is_pre_election: true
I20240501 08:12:47.150089 29370 tablet_bootstrap.cc:492] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db: No bootstrap required, opened a new log
I20240501 08:12:47.150252 29370 ts_tablet_manager.cc:1367] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240501 08:12:47.150328 29369 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255: No bootstrap required, opened a new log
W20240501 08:12:47.150385 29137 leader_election.cc:341] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:47.150501 29369 ts_tablet_manager.cc:1367] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:47.151041 29370 raft_consensus.cc:348] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.151180 29369 raft_consensus.cc:348] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.151300 29370 raft_consensus.cc:374] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.151381 29369 raft_consensus.cc:374] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.151430 29370 raft_consensus.cc:729] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc30284a7e44aa9f9b18e530a7d7db, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.151473 29369 raft_consensus.cc:729] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8bea7295884bc8bdc0d1c689469255, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.151710 29369 consensus_queue.cc:260] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.151715 29370 consensus_queue.cc:260] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.152372 29370 ts_tablet_manager.cc:1398] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:47.152493 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" is_pre_election: true
I20240501 08:12:47.152665 29370 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db: Bootstrap starting.
W20240501 08:12:47.152951 29138 leader_election.cc:341] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:47.153055 29138 leader_election.cc:302] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5; no voters: 86bc30284a7e44aa9f9b18e530a7d7db, ae8bea7295884bc8bdc0d1c689469255
I20240501 08:12:47.153275 29374 raft_consensus.cc:2726] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:47.154426 29370 tablet_bootstrap.cc:654] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:47.155242 29368 tablet_bootstrap.cc:492] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5: No bootstrap required, opened a new log
I20240501 08:12:47.155508 29368 ts_tablet_manager.cc:1367] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.007s
I20240501 08:12:47.156287 29368 raft_consensus.cc:348] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.156476 29368 raft_consensus.cc:374] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.156563 29368 raft_consensus.cc:729] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea04874bc846c1ac55ded6f9df08a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.156816 29368 consensus_queue.cc:260] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.157392 29368 ts_tablet_manager.cc:1398] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:47.158002 29370 tablet_bootstrap.cc:492] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db: No bootstrap required, opened a new log
I20240501 08:12:47.158177 29370 ts_tablet_manager.cc:1367] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:12:47.158823 29370 raft_consensus.cc:348] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.159176 29370 raft_consensus.cc:374] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:47.159286 29370 raft_consensus.cc:729] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc30284a7e44aa9f9b18e530a7d7db, State: Initialized, Role: FOLLOWER
I20240501 08:12:47.159590 29370 consensus_queue.cc:260] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.160333 29370 ts_tablet_manager.cc:1398] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240501 08:12:47.163385 29200 tablet.cc:2369] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:47.178254 29374 raft_consensus.cc:482] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:47.178553 29374 raft_consensus.cc:504] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.178961 29375 raft_consensus.cc:482] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:47.179100 29375 raft_consensus.cc:504] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.179311 29374 leader_election.cc:288] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603), 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:47.179526 29369 ts_tablet_manager.cc:1398] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255: Time spent starting tablet: real 0.029s user 0.020s sys 0.008s
I20240501 08:12:47.180078 29375 leader_election.cc:288] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213), 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:47.180145 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7b8f097fe54bdab219a389e622c587" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" is_pre_election: true
I20240501 08:12:47.180366 29323 raft_consensus.cc:2454] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea04874bc846c1ac55ded6f9df08a5 in term 0.
I20240501 08:12:47.180920 29138 leader_election.cc:302] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, 86bc30284a7e44aa9f9b18e530a7d7db; no voters:
I20240501 08:12:47.181454 29374 raft_consensus.cc:2781] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:47.181528 29374 raft_consensus.cc:482] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:47.181612 29374 raft_consensus.cc:3037] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.184412 29374 raft_consensus.cc:504] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.184486 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7b8f097fe54bdab219a389e622c587" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bea7295884bc8bdc0d1c689469255" is_pre_election: true
I20240501 08:12:47.184784 29249 raft_consensus.cc:2454] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea04874bc846c1ac55ded6f9df08a5 in term 0.
I20240501 08:12:47.185320 29374 leader_election.cc:288] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 election: Requested vote from peers ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603), 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:47.185609 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7b8f097fe54bdab219a389e622c587" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bea7295884bc8bdc0d1c689469255"
I20240501 08:12:47.185762 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7b8f097fe54bdab219a389e622c587" candidate_uuid: "4bea04874bc846c1ac55ded6f9df08a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc30284a7e44aa9f9b18e530a7d7db"
I20240501 08:12:47.185930 29323 raft_consensus.cc:3037] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.185768 29249 raft_consensus.cc:3037] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.188124 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "ae8bea7295884bc8bdc0d1c689469255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea04874bc846c1ac55ded6f9df08a5" is_pre_election: true
I20240501 08:12:47.188292 29174 raft_consensus.cc:2454] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8bea7295884bc8bdc0d1c689469255 in term 0.
I20240501 08:12:47.188686 29214 leader_election.cc:302] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, ae8bea7295884bc8bdc0d1c689469255; no voters:
I20240501 08:12:47.188781 29323 raft_consensus.cc:2454] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea04874bc846c1ac55ded6f9df08a5 in term 1.
I20240501 08:12:47.188885 29377 raft_consensus.cc:482] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:47.189038 29385 raft_consensus.cc:2781] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:47.189101 29385 raft_consensus.cc:482] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:47.188999 29377 raft_consensus.cc:504] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.189159 29385 raft_consensus.cc:3037] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.189124 29138 leader_election.cc:302] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, 86bc30284a7e44aa9f9b18e530a7d7db; no voters:
I20240501 08:12:47.189352 29374 raft_consensus.cc:2781] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:47.189709 29374 raft_consensus.cc:686] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea04874bc846c1ac55ded6f9df08a5, State: Running, Role: LEADER
I20240501 08:12:47.189790 29377 leader_election.cc:288] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213), ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603)
I20240501 08:12:47.190217 29374 consensus_queue.cc:237] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.188782 29249 raft_consensus.cc:2454] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea04874bc846c1ac55ded6f9df08a5 in term 1.
I20240501 08:12:47.192031 29385 raft_consensus.cc:504] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.194411 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "ae8bea7295884bc8bdc0d1c689469255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" is_pre_election: true
I20240501 08:12:47.194587 29323 raft_consensus.cc:2454] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8bea7295884bc8bdc0d1c689469255 in term 0.
I20240501 08:12:47.193173 29071 catalog_manager.cc:5484] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1). New cstate: current_term: 1 leader_uuid: "4bea04874bc846c1ac55ded6f9df08a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.195045 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "ae8bea7295884bc8bdc0d1c689469255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea04874bc846c1ac55ded6f9df08a5"
I20240501 08:12:47.195261 29174 raft_consensus.cc:3037] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.195693 29385 leader_election.cc:288] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [CANDIDATE]: Term 1 election: Requested vote from peers 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213), 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
I20240501 08:12:47.196251 29323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7495cdffb96945acba22f8450c9917ff" candidate_uuid: "ae8bea7295884bc8bdc0d1c689469255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc30284a7e44aa9f9b18e530a7d7db"
I20240501 08:12:47.196405 29323 raft_consensus.cc:3037] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.198786 29174 raft_consensus.cc:2454] T 7495cdffb96945acba22f8450c9917ff P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8bea7295884bc8bdc0d1c689469255 in term 1.
I20240501 08:12:47.198817 29323 raft_consensus.cc:2454] T 7495cdffb96945acba22f8450c9917ff P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8bea7295884bc8bdc0d1c689469255 in term 1.
I20240501 08:12:47.199218 29214 leader_election.cc:302] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, ae8bea7295884bc8bdc0d1c689469255; no voters:
I20240501 08:12:47.199635 29385 raft_consensus.cc:2781] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:47.199767 29385 raft_consensus.cc:686] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [term 1 LEADER]: Becoming Leader. State: Replica: ae8bea7295884bc8bdc0d1c689469255, State: Running, Role: LEADER
I20240501 08:12:47.200040 29385 consensus_queue.cc:237] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.202726 29071 catalog_manager.cc:5484] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2). New cstate: current_term: 1 leader_uuid: "ae8bea7295884bc8bdc0d1c689469255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.203701 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944204304b9a46b4a46788a5f6dd23a6" candidate_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea04874bc846c1ac55ded6f9df08a5" is_pre_election: true
I20240501 08:12:47.203876 29174 raft_consensus.cc:2454] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc30284a7e44aa9f9b18e530a7d7db in term 0.
I20240501 08:12:47.204286 29288 leader_election.cc:302] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, 86bc30284a7e44aa9f9b18e530a7d7db; no voters:
I20240501 08:12:47.204521 29377 raft_consensus.cc:2781] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:47.204608 29377 raft_consensus.cc:482] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:47.204684 29377 raft_consensus.cc:3037] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.205497 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944204304b9a46b4a46788a5f6dd23a6" candidate_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bea7295884bc8bdc0d1c689469255" is_pre_election: true
I20240501 08:12:47.205689 29249 raft_consensus.cc:2454] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc30284a7e44aa9f9b18e530a7d7db in term 0.
I20240501 08:12:47.207024 29377 raft_consensus.cc:504] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.207481 29377 leader_election.cc:288] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [CANDIDATE]: Term 1 election: Requested vote from peers 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213), ae8bea7295884bc8bdc0d1c689469255 (127.23.244.2:46603)
I20240501 08:12:47.207804 29174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944204304b9a46b4a46788a5f6dd23a6" candidate_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea04874bc846c1ac55ded6f9df08a5"
I20240501 08:12:47.207857 29249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944204304b9a46b4a46788a5f6dd23a6" candidate_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bea7295884bc8bdc0d1c689469255"
I20240501 08:12:47.207969 29174 raft_consensus.cc:3037] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.208021 29249 raft_consensus.cc:3037] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:47.210547 29249 raft_consensus.cc:2454] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc30284a7e44aa9f9b18e530a7d7db in term 1.
I20240501 08:12:47.210547 29174 raft_consensus.cc:2454] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc30284a7e44aa9f9b18e530a7d7db in term 1.
I20240501 08:12:47.210950 29288 leader_election.cc:302] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea04874bc846c1ac55ded6f9df08a5, 86bc30284a7e44aa9f9b18e530a7d7db; no voters:
I20240501 08:12:47.211177 29377 raft_consensus.cc:2781] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:47.211548 29377 raft_consensus.cc:686] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 LEADER]: Becoming Leader. State: Replica: 86bc30284a7e44aa9f9b18e530a7d7db, State: Running, Role: LEADER
I20240501 08:12:47.211776 29377 consensus_queue.cc:237] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } }
I20240501 08:12:47.214102 29071 catalog_manager.cc:5484] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3). New cstate: current_term: 1 leader_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.251237 24528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:47.254154 29395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:47.254885 29396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:47.255380 29398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:47.256141 24528 server_base.cc:1034] running on GCE node
I20240501 08:12:47.256357 24528 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:47.256428 24528 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:47.256492 24528 hybrid_clock.cc:648] HybridClock initialized: now 1714551167256473 us; error 0 us; skew 500 ppm
I20240501 08:12:47.256662 24528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:47.257994 24528 webserver.cc:460] Webserver started at http://127.23.244.4:36645/ using document root <none> and password file <none>
I20240501 08:12:47.258316 24528 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:47.258416 24528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:47.258553 24528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:47.259248 24528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-3-root/instance:
uuid: "00d4f2712ecd41a68f8fc0e6bab523d7"
format_stamp: "Formatted at 2024-05-01 08:12:47 on dist-test-slave-fck4"
I20240501 08:12:47.261870 24528 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 08:12:47.263540 29403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:47.263949 24528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:12:47.264075 24528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-3-root
uuid: "00d4f2712ecd41a68f8fc0e6bab523d7"
format_stamp: "Formatted at 2024-05-01 08:12:47 on dist-test-slave-fck4"
I20240501 08:12:47.264206 24528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskABejAt/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714551036644763-24528-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:47.276611 24528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:47.277208 24528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:47.277794 24528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:47.278679 24528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:47.278753 24528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:47.278868 24528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:47.278932 24528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:47.292603 29374 consensus_queue.cc:1035] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.292824 24528 rpc_server.cc:307] RPC server started. Bound to: 127.23.244.4:43723
I20240501 08:12:47.292851 29465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.244.4:43723 every 8 connection(s)
W20240501 08:12:47.295686 29388 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240501 08:12:47.295794 29388 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240501 08:12:47.298383 29374 consensus_queue.cc:1035] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.299090 29385 consensus_queue.cc:1035] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.303423 29466 heartbeater.cc:347] Connected to a master server at 127.23.244.62:43427
I20240501 08:12:47.303598 29466 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:47.304119 29466 heartbeater.cc:511] Master 127.23.244.62:43427 requested a full tablet report, sending...
I20240501 08:12:47.304891 29072 ts_manager.cc:194] Registered new tserver with Master: 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4:43723)
I20240501 08:12:47.305779 29072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39130
I20240501 08:12:47.317203 29377 consensus_queue.cc:1035] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.317239 29385 consensus_queue.cc:1035] T 7495cdffb96945acba22f8450c9917ff P ae8bea7295884bc8bdc0d1c689469255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.329749 29377 consensus_queue.cc:1035] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:47.840142 29125 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:12:47.841149 29125 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:12:47.841289 29125 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:12:47.847878 29323 consensus_queue.cc:237] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } }
I20240501 08:12:47.850025 29174 raft_consensus.cc:1261] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc30284a7e44aa9f9b18e530a7d7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.850790 29393 consensus_queue.cc:1035] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.854379 29174 raft_consensus.cc:2932] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
I20240501 08:12:47.858424 29249 raft_consensus.cc:1261] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Refusing update from remote peer 86bc30284a7e44aa9f9b18e530a7d7db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.858942 29483 consensus_queue.cc:1035] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.859063 29072 catalog_manager.cc:5484] T 944204304b9a46b4a46788a5f6dd23a6 P 4bea04874bc846c1ac55ded6f9df08a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
I20240501 08:12:47.860208 29249 raft_consensus.cc:2932] T 944204304b9a46b4a46788a5f6dd23a6 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
W20240501 08:12:47.865625 29286 consensus_peers.cc:485] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db -> Peer 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4:43723): Couldn't send request to peer 00d4f2712ecd41a68f8fc0e6bab523d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944204304b9a46b4a46788a5f6dd23a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.931015 29393 raft_consensus.cc:2932] T 944204304b9a46b4a46788a5f6dd23a6 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } attrs { replace: true } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
I20240501 08:12:47.940296 29174 consensus_queue.cc:237] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } }
I20240501 08:12:47.942741 29323 raft_consensus.cc:1261] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Refusing update from remote peer 4bea04874bc846c1ac55ded6f9df08a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.942741 29249 raft_consensus.cc:1261] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Refusing update from remote peer 4bea04874bc846c1ac55ded6f9df08a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.943367 29491 consensus_queue.cc:1035] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.943706 29374 consensus_queue.cc:1035] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.946354 29323 raft_consensus.cc:2932] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
I20240501 08:12:47.951684 29071 catalog_manager.cc:5484] T af7b8f097fe54bdab219a389e622c587 P 86bc30284a7e44aa9f9b18e530a7d7db reported cstate change: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New cstate: current_term: 1 leader_uuid: "4bea04874bc846c1ac55ded6f9df08a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
I20240501 08:12:47.957383 29492 ts_tablet_manager.cc:902] T 944204304b9a46b4a46788a5f6dd23a6 P 00d4f2712ecd41a68f8fc0e6bab523d7: Initiating tablet copy from peer 86bc30284a7e44aa9f9b18e530a7d7db (127.23.244.3:39221)
F20240501 08:12:47.957782 29492 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.queue_time_us, tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.queue_time_us) (tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.run_cpu_time_us, tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.run_cpu_time_us) (tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.run_wall_time_us, tablet-download-pool-21c5ff99f3f94039a7ba50580e56f0ff.run_wall_time_us) (tablet-download-pool-221cb4dc792649008eb83c64492ef820.queue_time_us, tablet-download-pool-221cb4dc792649008eb83c64492ef820.queue_time_us) (tablet-download-pool-221cb4dc792649008eb83c64492ef820.run_cpu_time_us, tablet-download-pool-221cb4dc792649008eb83c64492ef820.run_cpu_time_us) (tablet-download-pool-221cb4dc792649008eb83c64492ef820.run_wall_time_us, tablet-download-pool-221cb4dc792649008eb83c64492ef820.run_wall_time_us) (tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.queue_time_us, tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.queue_time_us) (tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.run_cpu_time_us, tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.run_cpu_time_us) (tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.run_wall_time_us, tablet-download-pool-25deaffcb0e441e590d7b300e1dcda69.run_wall_time_us) (tablet-download-pool-40396f8dc9614e369ce570cf51305386.queue_time_us, tablet-download-pool-40396f8dc9614e369ce570cf51305386.queue_time_us) (tablet-download-pool-40396f8dc9614e369ce570cf51305386.run_cpu_time_us, tablet-download-pool-40396f8dc9614e369ce570cf51305386.run_cpu_time_us) (tablet-download-pool-40396f8dc9614e369ce570cf51305386.run_wall_time_us, tablet-download-pool-40396f8dc9614e369ce570cf51305386.run_wall_time_us) (tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.queue_time_us, tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.queue_time_us) (tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.run_cpu_time_us, tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.run_cpu_time_us) (tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.run_wall_time_us, tablet-download-pool-4e3e31e67e264e9f8f9d07dbc2d90f61.run_wall_time_us) (tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.queue_time_us, tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.queue_time_us) (tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.run_cpu_time_us, tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.run_cpu_time_us) (tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.run_wall_time_us, tablet-download-pool-577a75e4e77d46f79a12ddcb864ebfae.run_wall_time_us) (tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.queue_time_us, tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.queue_time_us) (tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.run_cpu_time_us, tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.run_cpu_time_us) (tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.run_wall_time_us, tablet-download-pool-7af4b37f38284e8d9cdaaa56247b96e3.run_wall_time_us) (tablet-download-pool-9121766525b940d58127de0d7b9f5d57.queue_time_us, tablet-download-pool-9121766525b940d58127de0d7b9f5d57.queue_time_us) (tablet-download-pool-9121766525b940d58127de0d7b9f5d57.run_cpu_time_us, tablet-download-pool-9121766525b940d58127de0d7b9f5d57.run_cpu_time_us) (tablet-download-pool-9121766525b940d58127de0d7b9f5d57.run_wall_time_us, tablet-download-pool-9121766525b940d58127de0d7b9f5d57.run_wall_time_us) (tablet-download-pool-944204304b9a46b4a46788a5f6dd23a6.queue_time_us, tablet-download-pool-944204304b9a46b4a46788a5f6dd23a6.queue_time_us) (tablet-download-pool-947f432452be499595d4472aeb4a6bbe.queue_time_us, tablet-download-pool-947f432452be499595d4472aeb4a6bbe.queue_time_us) (tablet-download-pool-947f432452be499595d4472aeb4a6bbe.run_cpu_time_us, tablet-download-pool-947f432452be499595d4472aeb4a6bbe.run_cpu_time_us) (tablet-download-pool-947f432452be499595d4472aeb4a6bbe.run_wall_time_us, tablet-download-pool-947f432452be499595d4472aeb4a6bbe.run_wall_time_us) (tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.queue_time_us, tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.queue_time_us) (tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.run_cpu_time_us, tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.run_cpu_time_us) (tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.run_wall_time_us, tablet-download-pool-9a506a24817641c8afbfe3f8a5e696c0.run_wall_time_us) (tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.queue_time_us, tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.queue_time_us) (tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.run_cpu_time_us, tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.run_cpu_time_us) (tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.run_wall_time_us, tablet-download-pool-bde63d1ffa614078a4bbf3f46da72e8b.run_wall_time_us) (tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.queue_time_us, tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.queue_time_us) (tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.run_cpu_time_us, tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.run_cpu_time_us) (tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.run_wall_time_us, tablet-download-pool-df29fad2de5b4ac18e52d233ee63a6d0.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
W20240501 08:12:47.958202 29137 consensus_peers.cc:485] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 -> Peer 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4:43723): Couldn't send request to peer 00d4f2712ecd41a68f8fc0e6bab523d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af7b8f097fe54bdab219a389e622c587. This is attempt 1: this message will repeat every 5th retry.
*** Aborted at 1714551167 (unix time) try "date -d @1714551167" if you are using GNU date ***
I20240501 08:12:47.958509 29249 raft_consensus.cc:2932] T af7b8f097fe54bdab219a389e622c587 P ae8bea7295884bc8bdc0d1c689469255 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005fd0) received by PID 24528 (TID 0x7f28b41e4700) from PID 24528; stack trace: ***
@ 0x7f28e7cd1980 (unknown) at ??:0
@ 0x7f28dd01efb7 gsignal at ??:0
@ 0x7f28dd020921 abort at ??:0
@ 0x7f28dfad3dcd google::LogMessage::Fail() at ??:0
@ 0x7f28dfad7b93 google::LogMessage::SendToLog() at ??:0
I20240501 08:12:47.990041 29374 raft_consensus.cc:2932] T af7b8f097fe54bdab219a389e622c587 P 4bea04874bc846c1ac55ded6f9df08a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00d4f2712ecd41a68f8fc0e6bab523d7 (127.23.244.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4bea04874bc846c1ac55ded6f9df08a5" member_type: VOTER last_known_addr { host: "127.23.244.1" port: 41213 } } peers { permanent_uuid: "ae8bea7295884bc8bdc0d1c689469255" member_type: VOTER last_known_addr { host: "127.23.244.2" port: 46603 } } peers { permanent_uuid: "86bc30284a7e44aa9f9b18e530a7d7db" member_type: VOTER last_known_addr { host: "127.23.244.3" port: 39221 } attrs { replace: true } } peers { permanent_uuid: "00d4f2712ecd41a68f8fc0e6bab523d7" member_type: NON_VOTER last_known_addr { host: "127.23.244.4" port: 43723 } attrs { promote: true } } }
@ 0x7f28dfad37cc google::LogMessage::Flush() at ??:0
@ 0x7f28dfad4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28e1150742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f28e111cf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f28e1116cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f28f05e28de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f28f05fadaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f28f0725480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f28f075da8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:12:48.039376 29509 ts_tablet_manager.cc:902] T af7b8f097fe54bdab219a389e622c587 P 00d4f2712ecd41a68f8fc0e6bab523d7: Initiating tablet copy from peer 4bea04874bc846c1ac55ded6f9df08a5 (127.23.244.1:41213)
@ 0x7f28f0740981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f28f074070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28f55e298d std::function<>::operator()() at ??:0
@ 0x7f28e1120b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f28e112419c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f28e1124019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28f55e298d std::function<>::operator()() at ??:0
@ 0x7f28e10ff01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28e7cc66db start_thread at ??:0
@ 0x7f28dd10171f clone at ??:0