[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:22.845717 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:38679,127.26.178.125:46255,127.26.178.124:39263
I20240503 02:11:22.847632 27337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:22.848886 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:22.860644 27342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:22.861398 27343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:22.861691 27345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:22.862728 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:22.863627 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:22.863806 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:22.863891 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702282863870 us; error 0 us; skew 500 ppm
I20240503 02:11:22.864267 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:22.867085 27337 webserver.cc:460] Webserver started at http://127.26.178.126:35215/ using document root <none> and password file <none>
I20240503 02:11:22.867816 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:22.867918 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:22.868294 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:22.871788 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "b69d55fb671642f28b872b3dde45d3e8"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:22.877885 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:22.881659 27350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:22.883412 27337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:22.883601 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "b69d55fb671642f28b872b3dde45d3e8"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:22.883817 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:22.912987 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:22.913867 27337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:22.914180 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:22.933354 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:38679
I20240503 02:11:22.933384 27402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:38679 every 8 connection(s)
I20240503 02:11:22.935587 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:22.938614 27405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:22.939359 27406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:22.939600 27403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:11:22.940079 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:22.940632 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:22.940966 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:22.941056 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:22.941145 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702282941105 us; error 0 us; skew 500 ppm
I20240503 02:11:22.941352 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:22.942986 27337 webserver.cc:460] Webserver started at http://127.26.178.125:43559/ using document root <none> and password file <none>
I20240503 02:11:22.943244 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:22.943364 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:22.943521 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:22.944342 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root/instance:
uuid: "385ee22e179f436e8fed2e6f7c60f86c"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:22.947827 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:22.950048 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:22.950493 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:22.950620 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
uuid: "385ee22e179f436e8fed2e6f7c60f86c"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:22.950798 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:22.954160 27403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:22.969048 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:22.969915 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:22.972474 27403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:11:22.973766 27353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.178.125:46255: connect: Connection refused (error 111)
W20240503 02:11:22.976106 27403 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.26.178.125:46255: Network error: Client connection negotiation failed: client connection to 127.26.178.125:46255: connect: Connection refused (error 111)
I20240503 02:11:22.986034 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.125:46255
I20240503 02:11:22.986071 27468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.125:46255 every 8 connection(s)
I20240503 02:11:22.987488 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:22.987785 27469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:11:22.990847 27471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:22.991688 27472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:22.992058 27474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:22.992254 27469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:22.993140 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:22.993500 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:22.993595 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:22.993696 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702282993655 us; error 0 us; skew 500 ppm
I20240503 02:11:22.993858 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:22.995337 27337 webserver.cc:460] Webserver started at http://127.26.178.124:43587/ using document root <none> and password file <none>
I20240503 02:11:22.995586 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:22.995666 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:22.995767 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:22.996558 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root/instance:
uuid: "09842961d87e4b2686bb8d801c1a1387"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:23.000135 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:11:23.000384 27469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:23.002549 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.003019 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20240503 02:11:23.003158 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
uuid: "09842961d87e4b2686bb8d801c1a1387"
format_stamp: "Formatted at 2024-05-03 02:11:22 on dist-test-slave-0cbc"
I20240503 02:11:23.003319 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.007867 27469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:11:23.010020 27469 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.26.178.124:39263: Network error: Client connection negotiation failed: client connection to 127.26.178.124:39263: connect: Connection refused (error 111)
I20240503 02:11:23.017324 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.017905 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.025605 27403 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } attempt: 1
I20240503 02:11:23.033444 27403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:23.035260 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.124:39263
I20240503 02:11:23.035377 27533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.124:39263 every 8 connection(s)
I20240503 02:11:23.036571 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:11:23.037281 27535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:23.040010 27535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:23.047075 27535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:23.047885 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: Bootstrap starting.
I20240503 02:11:23.052584 27403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.053388 27535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:23.053987 27403 log.cc:824] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:23.057580 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: No bootstrap required, opened a new log
I20240503 02:11:23.062237 27469 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } attempt: 1
I20240503 02:11:23.063777 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: Bootstrap starting.
I20240503 02:11:23.068328 27535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.068760 27403 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.069283 27403 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.069478 27403 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69d55fb671642f28b872b3dde45d3e8, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.070900 27403 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.072410 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.072867 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.074558 27403 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.075382 27469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: Bootstrap starting.
I20240503 02:11:23.076853 27469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.078971 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: No bootstrap required, opened a new log
I20240503 02:11:23.080039 27535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.080477 27535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.080600 27535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09842961d87e4b2686bb8d801c1a1387, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.080971 27535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.082559 27535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.084806 27548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.085275 27548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.088475 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 02:11:23.089017 27469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: No bootstrap required, opened a new log
I20240503 02:11:23.089908 27469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.090137 27469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.090222 27469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385ee22e179f436e8fed2e6f7c60f86c, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.090523 27469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.091493 27555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.091856 27469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.091884 27555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:11:23.096953 27554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:23.097208 27554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:11:23.101676 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 02:11:23.103294 27571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:23.103403 27571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:11:23.107066 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:23.107678 27578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:23.107802 27578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:11:23.111814 27579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.112197 27580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.113168 27582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.114095 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:23.114409 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:23.114517 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:23.114585 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702283114562 us; error 0 us; skew 500 ppm
I20240503 02:11:23.114837 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.116452 27337 webserver.cc:460] Webserver started at http://127.26.178.65:40983/ using document root <none> and password file <none>
I20240503 02:11:23.116712 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.116798 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.116974 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.117774 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "e12bb5f2674f4bbb922b3db62fd88320"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.120803 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:23.122922 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.123363 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:23.123473 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "e12bb5f2674f4bbb922b3db62fd88320"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.123610 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.145253 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.145941 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.147015 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:23.148656 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:23.148777 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.148906 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:23.148993 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20240503 02:11:23.166563 27356 tablet.cc:2369] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:23.174475 27649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:43199 every 8 connection(s)
I20240503 02:11:23.174475 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:43199
I20240503 02:11:23.189169 27651 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38679
I20240503 02:11:23.189571 27651 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.190762 27651 heartbeater.cc:511] Master 127.26.178.126:38679 requested a full tablet report, sending...
I20240503 02:11:23.194134 27368 ts_manager.cc:194] Registered new tserver with Master: e12bb5f2674f4bbb922b3db62fd88320 (127.26.178.65:43199)
I20240503 02:11:23.189180 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:23.203402 27652 heartbeater.cc:347] Connected to a master server at 127.26.178.125:46255
I20240503 02:11:23.203629 27652 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.204492 27652 heartbeater.cc:511] Master 127.26.178.125:46255 requested a full tablet report, sending...
W20240503 02:11:23.205564 27657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.206184 27433 ts_manager.cc:194] Registered new tserver with Master: e12bb5f2674f4bbb922b3db62fd88320 (127.26.178.65:43199)
I20240503 02:11:23.206815 27650 heartbeater.cc:347] Connected to a master server at 127.26.178.124:39263
I20240503 02:11:23.206966 27650 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.207687 27650 heartbeater.cc:511] Master 127.26.178.124:39263 requested a full tablet report, sending...
W20240503 02:11:23.208490 27658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.209071 27499 ts_manager.cc:194] Registered new tserver with Master: e12bb5f2674f4bbb922b3db62fd88320 (127.26.178.65:43199)
W20240503 02:11:23.213025 27660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.213450 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:23.213750 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:23.213827 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:23.213914 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702283213870 us; error 0 us; skew 500 ppm
I20240503 02:11:23.214077 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.215735 27337 webserver.cc:460] Webserver started at http://127.26.178.66:44387/ using document root <none> and password file <none>
I20240503 02:11:23.215977 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.216087 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.216228 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.216897 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "a12e0f9627ac46359fc25d20018e503e"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.219811 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:11:23.221719 27665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.222139 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:23.222285 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "a12e0f9627ac46359fc25d20018e503e"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.222448 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.243026 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.243867 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.244688 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:23.245908 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:23.245998 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.246115 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:23.246186 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.267040 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:44693
I20240503 02:11:23.267046 27728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:44693 every 8 connection(s)
I20240503 02:11:23.289522 27729 heartbeater.cc:347] Connected to a master server at 127.26.178.124:39263
I20240503 02:11:23.289744 27729 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.290552 27729 heartbeater.cc:511] Master 127.26.178.124:39263 requested a full tablet report, sending...
I20240503 02:11:23.292265 27499 ts_manager.cc:194] Registered new tserver with Master: a12e0f9627ac46359fc25d20018e503e (127.26.178.66:44693)
I20240503 02:11:23.294505 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:23.295949 27730 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38679
I20240503 02:11:23.296103 27730 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.296767 27730 heartbeater.cc:511] Master 127.26.178.126:38679 requested a full tablet report, sending...
I20240503 02:11:23.297976 27368 ts_manager.cc:194] Registered new tserver with Master: a12e0f9627ac46359fc25d20018e503e (127.26.178.66:44693)
I20240503 02:11:23.300257 27731 heartbeater.cc:347] Connected to a master server at 127.26.178.125:46255
I20240503 02:11:23.300411 27731 heartbeater.cc:464] Registering TS with master...
W20240503 02:11:23.300978 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.301216 27731 heartbeater.cc:511] Master 127.26.178.125:46255 requested a full tablet report, sending...
I20240503 02:11:23.302153 27433 ts_manager.cc:194] Registered new tserver with Master: a12e0f9627ac46359fc25d20018e503e (127.26.178.66:44693)
W20240503 02:11:23.307116 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.308784 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.312626 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:23.312999 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:23.313105 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:23.313174 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702283313149 us; error 0 us; skew 500 ppm
I20240503 02:11:23.313388 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.315045 27337 webserver.cc:460] Webserver started at http://127.26.178.67:35085/ using document root <none> and password file <none>
I20240503 02:11:23.315343 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.315454 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.315624 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.316511 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.320093 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:11:23.322454 27744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.322969 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:23.323100 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-0cbc"
I20240503 02:11:23.323248 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.371421 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.372198 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.372840 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:23.373848 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:23.373952 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.374076 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:23.374161 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.394538 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:43731
I20240503 02:11:23.394563 27806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:43731 every 8 connection(s)
I20240503 02:11:23.411979 27808 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38679
I20240503 02:11:23.412222 27808 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.412868 27808 heartbeater.cc:511] Master 127.26.178.126:38679 requested a full tablet report, sending...
I20240503 02:11:23.414172 27368 ts_manager.cc:194] Registered new tserver with Master: ef1c3b0aaf974a5b95cb7e0091a8690c (127.26.178.67:43731)
I20240503 02:11:23.415895 27540 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:23.416293 27540 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.418574 27809 heartbeater.cc:347] Connected to a master server at 127.26.178.125:46255
I20240503 02:11:23.418776 27809 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.418974 27540 leader_election.cc:288] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 385ee22e179f436e8fed2e6f7c60f86c (127.26.178.125:46255), 09842961d87e4b2686bb8d801c1a1387 (127.26.178.124:39263)
I20240503 02:11:23.419507 27809 heartbeater.cc:511] Master 127.26.178.125:46255 requested a full tablet report, sending...
I20240503 02:11:23.420742 27433 ts_manager.cc:194] Registered new tserver with Master: ef1c3b0aaf974a5b95cb7e0091a8690c (127.26.178.67:43731)
I20240503 02:11:23.419936 27443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69d55fb671642f28b872b3dde45d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385ee22e179f436e8fed2e6f7c60f86c" is_pre_election: true
I20240503 02:11:23.421730 27443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69d55fb671642f28b872b3dde45d3e8 in term 0.
I20240503 02:11:23.422210 27807 heartbeater.cc:347] Connected to a master server at 127.26.178.124:39263
I20240503 02:11:23.422466 27807 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:23.420986 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69d55fb671642f28b872b3dde45d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09842961d87e4b2686bb8d801c1a1387" is_pre_election: true
I20240503 02:11:23.422478 27353 leader_election.cc:302] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385ee22e179f436e8fed2e6f7c60f86c, b69d55fb671642f28b872b3dde45d3e8; no voters:
I20240503 02:11:23.422900 27509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69d55fb671642f28b872b3dde45d3e8 in term 0.
I20240503 02:11:23.423131 27540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:23.423219 27807 heartbeater.cc:511] Master 127.26.178.124:39263 requested a full tablet report, sending...
I20240503 02:11:23.423296 27540 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:23.423436 27540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.425983 27540 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.426612 27540 leader_election.cc:288] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [CANDIDATE]: Term 1 election: Requested vote from peers 385ee22e179f436e8fed2e6f7c60f86c (127.26.178.125:46255), 09842961d87e4b2686bb8d801c1a1387 (127.26.178.124:39263)
I20240503 02:11:23.427183 27443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69d55fb671642f28b872b3dde45d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385ee22e179f436e8fed2e6f7c60f86c"
I20240503 02:11:23.427381 27443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.427778 27499 ts_manager.cc:194] Registered new tserver with Master: ef1c3b0aaf974a5b95cb7e0091a8690c (127.26.178.67:43731)
I20240503 02:11:23.428442 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013426181s
I20240503 02:11:23.430013 27443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69d55fb671642f28b872b3dde45d3e8 in term 1.
I20240503 02:11:23.430454 27353 leader_election.cc:302] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385ee22e179f436e8fed2e6f7c60f86c, b69d55fb671642f28b872b3dde45d3e8; no voters:
I20240503 02:11:23.430506 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69d55fb671642f28b872b3dde45d3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09842961d87e4b2686bb8d801c1a1387"
I20240503 02:11:23.430707 27509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.430974 27540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:23.435067 27540 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 1 LEADER]: Becoming Leader. State: Replica: b69d55fb671642f28b872b3dde45d3e8, State: Running, Role: LEADER
I20240503 02:11:23.435307 27509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69d55fb671642f28b872b3dde45d3e8 in term 1.
I20240503 02:11:23.435916 27540 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } }
I20240503 02:11:23.439255 27815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69d55fb671642f28b872b3dde45d3e8. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.439788 27815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.440783 27817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:23.442809 27817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:23.451134 27443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 1 FOLLOWER]: Refusing update from remote peer b69d55fb671642f28b872b3dde45d3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:23.451134 27509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 1 FOLLOWER]: Refusing update from remote peer b69d55fb671642f28b872b3dde45d3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:23.452139 27540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:23.452427 27815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:23.458484 27548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69d55fb671642f28b872b3dde45d3e8. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.458976 27548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.460290 27555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69d55fb671642f28b872b3dde45d3e8. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.460606 27555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.462641 27819 mvcc.cc:205] Tried to move back new op lower bound from 7023420553004740608 to 7023420552958849024. Current Snapshot: MvccSnapshot[applied={T|T < 7023420553004740608}]
I20240503 02:11:23.468217 27555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.468730 27555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.471506 27548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.471894 27548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:23.473193 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.473542 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.473295 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b69d55fb671642f28b872b3dde45d3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b69d55fb671642f28b872b3dde45d3e8" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 38679 } } peers { permanent_uuid: "385ee22e179f436e8fed2e6f7c60f86c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 46255 } } peers { permanent_uuid: "09842961d87e4b2686bb8d801c1a1387" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 39263 } } }
I20240503 02:11:23.473718 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.475152 27817 catalog_manager.cc:1347] Generated new cluster ID: b277cf8dfd4e460cb255752ee1f8a970
I20240503 02:11:23.475268 27817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:23.513435 27817 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:23.515312 27817 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:23.533859 27817 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: Generated new TSK 0
I20240503 02:11:23.535370 27817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:24.105181 27571 catalog_manager.cc:1259] Loaded cluster ID: b277cf8dfd4e460cb255752ee1f8a970
I20240503 02:11:24.105285 27571 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: loading cluster ID for follower catalog manager: success
I20240503 02:11:24.107378 27571 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: acquiring CA information for follower catalog manager: success
I20240503 02:11:24.108704 27571 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:11:24.109010 27578 catalog_manager.cc:1259] Loaded cluster ID: b277cf8dfd4e460cb255752ee1f8a970
I20240503 02:11:24.109086 27578 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: loading cluster ID for follower catalog manager: success
I20240503 02:11:24.110899 27578 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: acquiring CA information for follower catalog manager: success
I20240503 02:11:24.111866 27578 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:11:24.198441 27368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53274
I20240503 02:11:24.300493 27368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53282
I20240503 02:11:24.416893 27368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53286
I20240503 02:11:25.087991 27553 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:25.200722 27651 heartbeater.cc:503] Master 127.26.178.126:38679 was elected leader, sending a full tablet report...
I20240503 02:11:25.301764 27730 heartbeater.cc:503] Master 127.26.178.126:38679 was elected leader, sending a full tablet report...
I20240503 02:11:25.418246 27808 heartbeater.cc:503] Master 127.26.178.126:38679 was elected leader, sending a full tablet report...
I20240503 02:11:25.591948 27337 test_util.cc:268] Using random seed: 1017173359
I20240503 02:11:25.628778 27368 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53290:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:11:25.667969 27615 tablet_service.cc:1467] Processing CreateTablet for tablet 7a664b1a27b64e13b50aee980ed32b98 (DEFAULT_TABLE table=test-workload [id=f3a951f85dcd4a71b51f11ab0f2f5b8e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:11:25.669101 27615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a664b1a27b64e13b50aee980ed32b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:25.672199 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8675217ec648a488753a6fbbb6b60d (DEFAULT_TABLE table=test-workload [id=f3a951f85dcd4a71b51f11ab0f2f5b8e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:11:25.672784 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8675217ec648a488753a6fbbb6b60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:25.675616 27614 tablet_service.cc:1467] Processing CreateTablet for tablet 7698e84e8ec947f6ac9652499822c547 (DEFAULT_TABLE table=test-workload [id=f3a951f85dcd4a71b51f11ab0f2f5b8e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:11:25.676142 27614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7698e84e8ec947f6ac9652499822c547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:25.678618 27694 tablet_service.cc:1467] Processing CreateTablet for tablet 34dff389016242699fdf593869f45f0b (DEFAULT_TABLE table=test-workload [id=f3a951f85dcd4a71b51f11ab0f2f5b8e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:11:25.679129 27694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34dff389016242699fdf593869f45f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:25.685256 27854 tablet_bootstrap.cc:492] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: Bootstrap starting.
I20240503 02:11:25.685863 27855 tablet_bootstrap.cc:492] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: Bootstrap starting.
I20240503 02:11:25.687737 27856 tablet_bootstrap.cc:492] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: Bootstrap starting.
I20240503 02:11:25.688522 27855 tablet_bootstrap.cc:654] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.688639 27854 tablet_bootstrap.cc:654] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.689992 27856 tablet_bootstrap.cc:654] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.691080 27855 tablet_bootstrap.cc:492] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: No bootstrap required, opened a new log
I20240503 02:11:25.691148 27854 tablet_bootstrap.cc:492] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: No bootstrap required, opened a new log
I20240503 02:11:25.691342 27854 ts_tablet_manager.cc:1367] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:25.691349 27855 ts_tablet_manager.cc:1367] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:25.692296 27854 raft_consensus.cc:348] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 43731 } }
I20240503 02:11:25.692447 27854 raft_consensus.cc:374] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.692560 27854 raft_consensus.cc:729] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1c3b0aaf974a5b95cb7e0091a8690c, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.692776 27854 consensus_queue.cc:260] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 43731 } }
I20240503 02:11:25.692857 27855 raft_consensus.cc:348] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.693006 27854 raft_consensus.cc:388] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:25.693113 27855 raft_consensus.cc:374] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.693114 27854 raft_consensus.cc:482] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:25.693262 27855 raft_consensus.cc:729] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12bb5f2674f4bbb922b3db62fd88320, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.693293 27854 raft_consensus.cc:3037] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.693526 27855 consensus_queue.cc:260] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.693773 27855 raft_consensus.cc:388] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:25.693883 27855 raft_consensus.cc:482] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:25.694023 27855 raft_consensus.cc:3037] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.695641 27854 raft_consensus.cc:504] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 43731 } }
I20240503 02:11:25.695853 27854 leader_election.cc:302] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef1c3b0aaf974a5b95cb7e0091a8690c; no voters:
I20240503 02:11:25.698328 27856 tablet_bootstrap.cc:492] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: No bootstrap required, opened a new log
I20240503 02:11:25.698452 27855 raft_consensus.cc:504] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.698530 27856 ts_tablet_manager.cc:1367] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240503 02:11:25.698872 27854 leader_election.cc:288] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:25.699050 27860 raft_consensus.cc:2781] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.699532 27856 raft_consensus.cc:348] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12e0f9627ac46359fc25d20018e503e" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44693 } }
I20240503 02:11:25.699770 27856 raft_consensus.cc:374] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.699883 27856 raft_consensus.cc:729] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12e0f9627ac46359fc25d20018e503e, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.700155 27856 consensus_queue.cc:260] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12e0f9627ac46359fc25d20018e503e" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44693 } }
I20240503 02:11:25.700410 27856 raft_consensus.cc:388] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:25.700506 27856 raft_consensus.cc:482] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:25.700634 27856 raft_consensus.cc:3037] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.703053 27860 raft_consensus.cc:686] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 1 LEADER]: Becoming Leader. State: Replica: ef1c3b0aaf974a5b95cb7e0091a8690c, State: Running, Role: LEADER
I20240503 02:11:25.703090 27855 leader_election.cc:302] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12bb5f2674f4bbb922b3db62fd88320; no voters:
I20240503 02:11:25.703840 27854 ts_tablet_manager.cc:1398] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20240503 02:11:25.704241 27860 consensus_queue.cc:237] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 43731 } }
I20240503 02:11:25.704452 27856 raft_consensus.cc:504] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12e0f9627ac46359fc25d20018e503e" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44693 } }
I20240503 02:11:25.704674 27856 leader_election.cc:302] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12e0f9627ac46359fc25d20018e503e; no voters:
I20240503 02:11:25.706737 27856 leader_election.cc:288] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:25.706866 27866 raft_consensus.cc:2781] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.709470 27855 leader_election.cc:288] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:25.709625 27863 raft_consensus.cc:2781] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.710961 27856 ts_tablet_manager.cc:1398] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20240503 02:11:25.713548 27866 raft_consensus.cc:686] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 1 LEADER]: Becoming Leader. State: Replica: a12e0f9627ac46359fc25d20018e503e, State: Running, Role: LEADER
I20240503 02:11:25.713954 27866 consensus_queue.cc:237] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12e0f9627ac46359fc25d20018e503e" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44693 } }
I20240503 02:11:25.715065 27855 ts_tablet_manager.cc:1398] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: Time spent starting tablet: real 0.023s user 0.009s sys 0.005s
I20240503 02:11:25.715216 27863 raft_consensus.cc:686] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 LEADER]: Becoming Leader. State: Replica: e12bb5f2674f4bbb922b3db62fd88320, State: Running, Role: LEADER
I20240503 02:11:25.715494 27863 consensus_queue.cc:237] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.715811 27855 tablet_bootstrap.cc:492] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: Bootstrap starting.
I20240503 02:11:25.717897 27855 tablet_bootstrap.cc:654] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.728787 27366 catalog_manager.cc:5484] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e reported cstate change: term changed from 0 to 1, leader changed from <none> to a12e0f9627ac46359fc25d20018e503e (127.26.178.66). New cstate: current_term: 1 leader_uuid: "a12e0f9627ac46359fc25d20018e503e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12e0f9627ac46359fc25d20018e503e" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44693 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:25.729400 27855 tablet_bootstrap.cc:492] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: No bootstrap required, opened a new log
I20240503 02:11:25.729590 27855 ts_tablet_manager.cc:1367] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240503 02:11:25.729413 27365 catalog_manager.cc:5484] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c reported cstate change: term changed from 0 to 1, leader changed from <none> to ef1c3b0aaf974a5b95cb7e0091a8690c (127.26.178.67). New cstate: current_term: 1 leader_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c3b0aaf974a5b95cb7e0091a8690c" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 43731 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:25.731297 27855 raft_consensus.cc:348] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.728258 27368 catalog_manager.cc:5484] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 reported cstate change: term changed from 0 to 1, leader changed from <none> to e12bb5f2674f4bbb922b3db62fd88320 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "e12bb5f2674f4bbb922b3db62fd88320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:25.731534 27855 raft_consensus.cc:374] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.731626 27855 raft_consensus.cc:729] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12bb5f2674f4bbb922b3db62fd88320, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.731868 27855 consensus_queue.cc:260] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.732043 27855 raft_consensus.cc:388] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:25.732184 27855 raft_consensus.cc:482] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:25.732316 27855 raft_consensus.cc:3037] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.735219 27855 raft_consensus.cc:504] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.735446 27855 leader_election.cc:302] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12bb5f2674f4bbb922b3db62fd88320; no voters:
I20240503 02:11:25.735656 27855 leader_election.cc:288] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:25.735826 27870 raft_consensus.cc:2781] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.736074 27870 raft_consensus.cc:686] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 LEADER]: Becoming Leader. State: Replica: e12bb5f2674f4bbb922b3db62fd88320, State: Running, Role: LEADER
I20240503 02:11:25.736407 27870 consensus_queue.cc:237] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } }
I20240503 02:11:25.736843 27855 ts_tablet_manager.cc:1398] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 02:11:25.751219 27366 catalog_manager.cc:5484] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 reported cstate change: term changed from 0 to 1, leader changed from <none> to e12bb5f2674f4bbb922b3db62fd88320 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "e12bb5f2674f4bbb922b3db62fd88320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bb5f2674f4bbb922b3db62fd88320" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 43199 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:25.759920 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:11:25.770320 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:25.770753 27337 tablet_replica.cc:330] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320: stopping tablet replica
I20240503 02:11:25.771098 27337 raft_consensus.cc:2229] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:25.771337 27337 raft_consensus.cc:2258] T 7a664b1a27b64e13b50aee980ed32b98 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.772755 27337 tablet_replica.cc:330] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320: stopping tablet replica
I20240503 02:11:25.772883 27337 raft_consensus.cc:2229] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:25.773006 27337 raft_consensus.cc:2258] T 7698e84e8ec947f6ac9652499822c547 P e12bb5f2674f4bbb922b3db62fd88320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.778517 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:11:25.784282 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:11:25.793711 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:25.794062 27337 tablet_replica.cc:330] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e: stopping tablet replica
I20240503 02:11:25.794266 27337 raft_consensus.cc:2229] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:25.794497 27337 raft_consensus.cc:2258] T 34dff389016242699fdf593869f45f0b P a12e0f9627ac46359fc25d20018e503e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.798934 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:11:25.801606 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:11:25.811132 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:25.811597 27337 tablet_replica.cc:330] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c: stopping tablet replica
I20240503 02:11:25.811810 27337 raft_consensus.cc:2229] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:25.811990 27337 raft_consensus.cc:2258] T 9e8675217ec648a488753a6fbbb6b60d P ef1c3b0aaf974a5b95cb7e0091a8690c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.826174 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:11:25.829064 27337 master.cc:537] Master@127.26.178.126:38679 shutting down...
I20240503 02:11:25.837093 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:25.837571 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.837720 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P b69d55fb671642f28b872b3dde45d3e8: stopping tablet replica
I20240503 02:11:25.853129 27337 master.cc:552] Master@127.26.178.126:38679 shutdown complete.
I20240503 02:11:25.858000 27337 master.cc:537] Master@127.26.178.125:46255 shutting down...
I20240503 02:11:25.865172 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:25.865401 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.865514 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 385ee22e179f436e8fed2e6f7c60f86c: stopping tablet replica
I20240503 02:11:25.879458 27337 master.cc:552] Master@127.26.178.125:46255 shutdown complete.
I20240503 02:11:25.883642 27337 master.cc:537] Master@127.26.178.124:39263 shutting down...
I20240503 02:11:25.890206 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:25.890427 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:25.890555 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 09842961d87e4b2686bb8d801c1a1387: stopping tablet replica
I20240503 02:11:25.904563 27337 master.cc:552] Master@127.26.178.124:39263 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3075 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 02:11:25.918542 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:45195
I20240503 02:11:25.919083 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.921826 27878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.922116 27879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.922583 27881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.923565 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:25.923823 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:25.923894 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:25.923955 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702285923936 us; error 0 us; skew 500 ppm
I20240503 02:11:25.924129 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.929934 27337 webserver.cc:460] Webserver started at http://127.26.178.126:37717/ using document root <none> and password file <none>
I20240503 02:11:25.930197 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.930279 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.930444 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.931211 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "517961dab66c4502be552554838b650e"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-0cbc"
I20240503 02:11:25.933763 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:25.935446 27886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.935787 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:25.935896 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "517961dab66c4502be552554838b650e"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-0cbc"
I20240503 02:11:25.936012 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.943727 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.944288 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.958774 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:45195
I20240503 02:11:25.958817 27937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:45195 every 8 connection(s)
I20240503 02:11:25.960220 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:25.964497 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e: Bootstrap starting.
I20240503 02:11:25.965839 27938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.967522 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e: No bootstrap required, opened a new log
I20240503 02:11:25.968199 27938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER }
I20240503 02:11:25.968343 27938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.968454 27938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517961dab66c4502be552554838b650e, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.968672 27938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER }
I20240503 02:11:25.968833 27938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:25.968919 27938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:25.969048 27938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.970909 27938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER }
I20240503 02:11:25.971158 27938 leader_election.cc:302] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 517961dab66c4502be552554838b650e; no voters:
I20240503 02:11:25.971606 27938 leader_election.cc:288] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:25.971747 27941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.972283 27941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 1 LEADER]: Becoming Leader. State: Replica: 517961dab66c4502be552554838b650e, State: Running, Role: LEADER
I20240503 02:11:25.972635 27938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:25.972568 27941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER }
I20240503 02:11:25.973634 27942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "517961dab66c4502be552554838b650e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER } }
I20240503 02:11:25.973678 27943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517961dab66c4502be552554838b650e. Latest consensus state: current_term: 1 leader_uuid: "517961dab66c4502be552554838b650e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517961dab66c4502be552554838b650e" member_type: VOTER } }
I20240503 02:11:25.973929 27942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:25.973963 27943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:25.975392 27948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:25.976871 27948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:25.978338 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:11:25.980165 27948 catalog_manager.cc:1347] Generated new cluster ID: 54957329a5fc4363a8245e17ccff6fce
I20240503 02:11:25.980250 27948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:26.008437 27948 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:26.009332 27948 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:26.026919 27948 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e: Generated new TSK 0
I20240503 02:11:26.027206 27948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:26.043380 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.046658 27959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.046929 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.048205 27962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.048578 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:26.048830 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:26.048899 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:26.048977 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702286048944 us; error 0 us; skew 500 ppm
I20240503 02:11:26.049136 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.050490 27337 webserver.cc:460] Webserver started at http://127.26.178.65:35739/ using document root <none> and password file <none>
I20240503 02:11:26.050769 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.050880 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.051020 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.051730 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.054701 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:26.056353 27967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.056725 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:26.056838 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.056977 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.073877 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:26.074481 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.075083 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.076051 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.076120 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.076249 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.076305 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.091955 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:35101
I20240503 02:11:26.091979 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:35101 every 8 connection(s)
I20240503 02:11:26.093628 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.097214 28034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.097546 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.101001 28030 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45195
W20240503 02:11:26.101024 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.101219 28030 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.101500 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:26.101792 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:11:26.101800 28030 heartbeater.cc:511] Master 127.26.178.126:45195 requested a full tablet report, sending...
W20240503 02:11:26.101893 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:26.101959 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702286101934 us; error 0 us; skew 500 ppm
I20240503 02:11:26.102137 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.102874 27903 ts_manager.cc:194] Registered new tserver with Master: e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101)
I20240503 02:11:26.103657 27337 webserver.cc:460] Webserver started at http://127.26.178.66:39623/ using document root <none> and password file <none>
I20240503 02:11:26.103937 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.104024 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.104176 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.104501 27903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46516
I20240503 02:11:26.104966 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "5b57ee6b7baf4e858fca86cdf3241585"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.107492 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:11:26.109009 28042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.109359 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:11:26.109467 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "5b57ee6b7baf4e858fca86cdf3241585"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.109592 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.122256 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:26.122867 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.123448 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.124235 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.124322 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.124424 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.124502 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.139196 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:42049
I20240503 02:11:26.139263 28104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:42049 every 8 connection(s)
I20240503 02:11:26.140903 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.144491 28109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.144485 28108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.146574 28111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.146963 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:26.147279 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:26.147373 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:26.147465 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702286147423 us; error 0 us; skew 500 ppm
I20240503 02:11:26.147632 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.148057 28105 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45195
I20240503 02:11:26.148226 28105 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.148700 28105 heartbeater.cc:511] Master 127.26.178.126:45195 requested a full tablet report, sending...
I20240503 02:11:26.149921 27337 webserver.cc:460] Webserver started at http://127.26.178.67:41321/ using document root <none> and password file <none>
I20240503 02:11:26.149983 27903 ts_manager.cc:194] Registered new tserver with Master: 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:26.150197 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.150314 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.150475 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.151105 27903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46518
I20240503 02:11:26.151422 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "289c6211929b44bcb758b1a9079fde00"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.154003 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:26.155773 28116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.156167 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:11:26.156292 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "289c6211929b44bcb758b1a9079fde00"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-0cbc"
I20240503 02:11:26.156436 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.165190 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:26.165760 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.166352 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.167287 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.167368 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.167495 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.167557 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.181640 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:40449
I20240503 02:11:26.181659 28178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:40449 every 8 connection(s)
I20240503 02:11:26.188658 28179 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45195
I20240503 02:11:26.188798 28179 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.189199 28179 heartbeater.cc:511] Master 127.26.178.126:45195 requested a full tablet report, sending...
I20240503 02:11:26.190042 27903 ts_manager.cc:194] Registered new tserver with Master: 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:26.190783 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008120479s
I20240503 02:11:26.190816 27903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46520
I20240503 02:11:27.105878 28030 heartbeater.cc:503] Master 127.26.178.126:45195 was elected leader, sending a full tablet report...
I20240503 02:11:27.152557 28105 heartbeater.cc:503] Master 127.26.178.126:45195 was elected leader, sending a full tablet report...
I20240503 02:11:27.192044 28179 heartbeater.cc:503] Master 127.26.178.126:45195 was elected leader, sending a full tablet report...
I20240503 02:11:27.218894 27337 test_util.cc:268] Using random seed: 1018800295
I20240503 02:11:27.233722 27903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46524:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:11:27.234503 27903 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:11:27.269913 27995 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff4d1a84e94055a2063802a103eb66 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:11:27.270542 27995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff4d1a84e94055a2063802a103eb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.271529 28144 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff4d1a84e94055a2063802a103eb66 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:11:27.271770 27994 tablet_service.cc:1467] Processing CreateTablet for tablet c719be39f5684041b2e1e5c116b96a2c (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:11:27.272063 28144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff4d1a84e94055a2063802a103eb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.272290 27994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c719be39f5684041b2e1e5c116b96a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.272634 28143 tablet_service.cc:1467] Processing CreateTablet for tablet c719be39f5684041b2e1e5c116b96a2c (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:11:27.273087 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c719be39f5684041b2e1e5c116b96a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.275107 28140 tablet_service.cc:1467] Processing CreateTablet for tablet e077f639cee044bf863f7c2382180644 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:11:27.275620 28140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077f639cee044bf863f7c2382180644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.276055 28142 tablet_service.cc:1467] Processing CreateTablet for tablet d94b14b51bad422b9d559b7b7a48855d (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:11:27.276520 28142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b14b51bad422b9d559b7b7a48855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.277632 27993 tablet_service.cc:1467] Processing CreateTablet for tablet d94b14b51bad422b9d559b7b7a48855d (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:11:27.278054 27993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b14b51bad422b9d559b7b7a48855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.279095 28139 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5b6c37226649e0865bde82646c1ce7 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:11:27.279523 28139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5b6c37226649e0865bde82646c1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.284287 28141 tablet_service.cc:1467] Processing CreateTablet for tablet 2102533fedbf43ac840cde53fcc060a6 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:11:27.286986 28141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2102533fedbf43ac840cde53fcc060a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.287046 28138 tablet_service.cc:1467] Processing CreateTablet for tablet 31bd947bec044d7d8ded1d301b33d12f (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:11:27.286341 28137 tablet_service.cc:1467] Processing CreateTablet for tablet 91f343ae46274a9da3ea761fbc614d1e (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:11:27.288473 27995 tablet_service.cc:1467] Processing CreateTablet for tablet 2102533fedbf43ac840cde53fcc060a6 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:11:27.288978 27995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2102533fedbf43ac840cde53fcc060a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.296522 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f343ae46274a9da3ea761fbc614d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.297634 28138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bd947bec044d7d8ded1d301b33d12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.300153 28070 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff4d1a84e94055a2063802a103eb66 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:11:27.300632 28070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff4d1a84e94055a2063802a103eb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.302553 27989 tablet_service.cc:1467] Processing CreateTablet for tablet 91f343ae46274a9da3ea761fbc614d1e (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:11:27.303740 27992 tablet_service.cc:1467] Processing CreateTablet for tablet e077f639cee044bf863f7c2382180644 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:11:27.304217 27992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077f639cee044bf863f7c2382180644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.302933 28069 tablet_service.cc:1467] Processing CreateTablet for tablet c719be39f5684041b2e1e5c116b96a2c (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:11:27.307452 27991 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5b6c37226649e0865bde82646c1ce7 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:11:27.309003 28069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c719be39f5684041b2e1e5c116b96a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.311872 27989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f343ae46274a9da3ea761fbc614d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.312486 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5b6c37226649e0865bde82646c1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.312835 27990 tablet_service.cc:1467] Processing CreateTablet for tablet 31bd947bec044d7d8ded1d301b33d12f (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:11:27.313803 28065 tablet_service.cc:1467] Processing CreateTablet for tablet 6e5b6c37226649e0865bde82646c1ce7 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:11:27.315357 28063 tablet_service.cc:1467] Processing CreateTablet for tablet 91f343ae46274a9da3ea761fbc614d1e (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:11:27.320214 28067 tablet_service.cc:1467] Processing CreateTablet for tablet 2102533fedbf43ac840cde53fcc060a6 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:11:27.315609 28066 tablet_service.cc:1467] Processing CreateTablet for tablet e077f639cee044bf863f7c2382180644 (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:11:27.321451 27990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bd947bec044d7d8ded1d301b33d12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.322311 28065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e5b6c37226649e0865bde82646c1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.322355 28200 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.322885 28064 tablet_service.cc:1467] Processing CreateTablet for tablet 31bd947bec044d7d8ded1d301b33d12f (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:11:27.325042 28067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2102533fedbf43ac840cde53fcc060a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.326311 28066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e077f639cee044bf863f7c2382180644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.326349 28068 tablet_service.cc:1467] Processing CreateTablet for tablet d94b14b51bad422b9d559b7b7a48855d (DEFAULT_TABLE table=test-workload [id=267d459d94644d9fb882979b6c1a25dd]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:11:27.331346 28199 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.334101 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f343ae46274a9da3ea761fbc614d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.337553 28064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bd947bec044d7d8ded1d301b33d12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.338219 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94b14b51bad422b9d559b7b7a48855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.340809 28201 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.360234 28199 tablet_bootstrap.cc:654] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.368045 28200 tablet_bootstrap.cc:654] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.369818 28201 tablet_bootstrap.cc:654] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.370396 28200 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.370563 28200 ts_tablet_manager.cc:1367] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240503 02:11:27.371397 28200 raft_consensus.cc:348] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.371515 28199 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.371606 28200 raft_consensus.cc:374] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.371733 28200 raft_consensus.cc:729] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.371737 28199 ts_tablet_manager.cc:1367] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20240503 02:11:27.372089 28200 consensus_queue.cc:260] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.372660 28199 raft_consensus.cc:348] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.372857 28199 raft_consensus.cc:374] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.372902 28200 ts_tablet_manager.cc:1398] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:27.372970 28199 raft_consensus.cc:729] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.373167 28200 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.373190 28199 consensus_queue.cc:260] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.373960 28199 ts_tablet_manager.cc:1398] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:27.374276 28199 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.374890 28200 tablet_bootstrap.cc:654] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.376099 28201 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.376255 28201 ts_tablet_manager.cc:1367] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240503 02:11:27.376572 28199 tablet_bootstrap.cc:654] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.377506 28201 raft_consensus.cc:348] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.377763 28201 raft_consensus.cc:374] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.377871 28201 raft_consensus.cc:729] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.378108 28201 consensus_queue.cc:260] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.379081 28199 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.379086 28201 ts_tablet_manager.cc:1398] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:11:27.379261 28199 ts_tablet_manager.cc:1367] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:27.379354 28200 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.379403 28201 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.379518 28200 ts_tablet_manager.cc:1367] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:11:27.380066 28199 raft_consensus.cc:348] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.380293 28199 raft_consensus.cc:374] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.380237 28200 raft_consensus.cc:348] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.380383 28199 raft_consensus.cc:729] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.380460 28200 raft_consensus.cc:374] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.380534 28200 raft_consensus.cc:729] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.380656 28199 consensus_queue.cc:260] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.380771 28200 consensus_queue.cc:260] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.381115 28201 tablet_bootstrap.cc:654] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.381247 28199 ts_tablet_manager.cc:1398] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:11:27.381325 28200 ts_tablet_manager.cc:1398] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:27.381568 28199 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.381639 28200 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.383227 28199 tablet_bootstrap.cc:654] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.383606 28200 tablet_bootstrap.cc:654] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.388265 28199 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.388427 28199 ts_tablet_manager.cc:1367] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 02:11:27.389111 28199 raft_consensus.cc:348] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.389276 28199 raft_consensus.cc:374] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.389392 28199 raft_consensus.cc:729] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.389533 28201 tablet_bootstrap.cc:492] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.389647 28199 consensus_queue.cc:260] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.389750 28201 ts_tablet_manager.cc:1367] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 02:11:27.390178 28199 ts_tablet_manager.cc:1398] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:27.390472 28199 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.390758 28201 raft_consensus.cc:348] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.391028 28201 raft_consensus.cc:374] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.391163 28201 raft_consensus.cc:729] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.391499 28201 consensus_queue.cc:260] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.392127 28201 ts_tablet_manager.cc:1398] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:27.392200 28199 tablet_bootstrap.cc:654] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.394301 28200 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.394450 28200 ts_tablet_manager.cc:1367] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240503 02:11:27.395246 28200 raft_consensus.cc:348] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.395466 28201 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.395468 28200 raft_consensus.cc:374] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.395617 28200 raft_consensus.cc:729] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.395870 28200 consensus_queue.cc:260] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.397195 28200 ts_tablet_manager.cc:1398] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:11:27.397442 28201 tablet_bootstrap.cc:654] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.397500 28200 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.399600 28201 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.399744 28201 ts_tablet_manager.cc:1367] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:11:27.400044 28200 tablet_bootstrap.cc:654] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.400570 28201 raft_consensus.cc:348] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.400794 28201 raft_consensus.cc:374] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.400897 28201 raft_consensus.cc:729] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.401212 28201 consensus_queue.cc:260] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.401786 28201 ts_tablet_manager.cc:1398] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:11:27.402097 28201 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.402592 28200 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.402858 28200 ts_tablet_manager.cc:1367] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:11:27.403890 28201 tablet_bootstrap.cc:654] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.404165 28200 raft_consensus.cc:348] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.404409 28200 raft_consensus.cc:374] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.404511 28200 raft_consensus.cc:729] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.404814 28200 consensus_queue.cc:260] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.405674 28199 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.405854 28199 ts_tablet_manager.cc:1367] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 02:11:27.405934 28200 ts_tablet_manager.cc:1398] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:27.406201 28200 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.406592 28199 raft_consensus.cc:348] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.406854 28199 raft_consensus.cc:374] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.406944 28199 raft_consensus.cc:729] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.407199 28199 consensus_queue.cc:260] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.407768 28200 tablet_bootstrap.cc:654] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.408051 28205 raft_consensus.cc:482] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.408213 28205 raft_consensus.cc:504] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.408500 28199 ts_tablet_manager.cc:1398] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:27.408805 28199 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.409627 28205 leader_election.cc:288] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:27.411099 28199 tablet_bootstrap.cc:654] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.416209 28200 tablet_bootstrap.cc:492] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.416394 28200 ts_tablet_manager.cc:1367] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:11:27.417412 28200 raft_consensus.cc:348] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.417698 28200 raft_consensus.cc:374] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.419739 28201 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.419925 28201 ts_tablet_manager.cc:1367] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:11:27.417819 28200 raft_consensus.cc:729] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.420262 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.420572 28005 raft_consensus.cc:2454] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 0.
I20240503 02:11:27.420675 28201 raft_consensus.cc:348] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.420610 28200 consensus_queue.cc:260] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.420876 28201 raft_consensus.cc:374] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.420962 28201 raft_consensus.cc:729] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.421288 28117 leader_election.cc:302] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.421237 28201 consensus_queue.cc:260] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.421444 28200 ts_tablet_manager.cc:1398] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:11:27.421562 28205 raft_consensus.cc:2781] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.424378 28205 raft_consensus.cc:482] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.424521 28200 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.424522 28205 raft_consensus.cc:3037] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.426731 28199 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.426883 28199 ts_tablet_manager.cc:1367] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.014s
I20240503 02:11:27.421753 28201 ts_tablet_manager.cc:1398] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:27.427511 28201 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.427810 28199 raft_consensus.cc:348] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.427995 28199 raft_consensus.cc:374] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.428078 28199 raft_consensus.cc:729] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.428153 28200 tablet_bootstrap.cc:654] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.428332 28199 consensus_queue.cc:260] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.428714 28205 raft_consensus.cc:504] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.429256 28201 tablet_bootstrap.cc:654] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.430698 28199 ts_tablet_manager.cc:1398] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:11:27.430997 28199 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.432412 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.432616 28005 raft_consensus.cc:3037] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.432667 28199 tablet_bootstrap.cc:654] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.436950 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
W20240503 02:11:27.438194 28120 leader_election.cc:341] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:27.439478 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:27.439818 28205 leader_election.cc:288] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
W20240503 02:11:27.440057 28120 leader_election.cc:341] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:27.440339 28005 raft_consensus.cc:2454] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 1.
I20240503 02:11:27.440863 28200 tablet_bootstrap.cc:492] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.440851 28117 leader_election.cc:302] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters: 5b57ee6b7baf4e858fca86cdf3241585
I20240503 02:11:27.441124 28200 ts_tablet_manager.cc:1367] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20240503 02:11:27.441545 28205 raft_consensus.cc:2781] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.442741 28205 raft_consensus.cc:686] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Becoming Leader. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Running, Role: LEADER
I20240503 02:11:27.443020 28199 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.443176 28199 ts_tablet_manager.cc:1367] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 02:11:27.443050 28205 consensus_queue.cc:237] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.443694 28201 tablet_bootstrap.cc:492] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.443845 28201 ts_tablet_manager.cc:1367] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20240503 02:11:27.443960 28199 raft_consensus.cc:348] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.444188 28199 raft_consensus.cc:374] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.444276 28199 raft_consensus.cc:729] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.444459 28200 raft_consensus.cc:348] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.444685 28200 raft_consensus.cc:374] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.444576 28199 consensus_queue.cc:260] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.444797 28200 raft_consensus.cc:729] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.445128 28199 ts_tablet_manager.cc:1398] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:27.445188 28200 consensus_queue.cc:260] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.445524 28199 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.446483 28200 ts_tablet_manager.cc:1398] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:11:27.446812 28200 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.447539 27903 catalog_manager.cc:5484] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289c6211929b44bcb758b1a9079fde00 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "289c6211929b44bcb758b1a9079fde00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:27.448222 28199 tablet_bootstrap.cc:654] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.448457 28200 tablet_bootstrap.cc:654] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.444795 28201 raft_consensus.cc:348] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.450069 28201 raft_consensus.cc:374] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.450277 28201 raft_consensus.cc:729] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.450635 28201 consensus_queue.cc:260] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.452334 28201 ts_tablet_manager.cc:1398] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:11:27.452762 28201 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.455097 28201 tablet_bootstrap.cc:654] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.457192 28200 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.457928 28199 tablet_bootstrap.cc:492] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.458098 28200 ts_tablet_manager.cc:1367] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 02:11:27.458182 28199 ts_tablet_manager.cc:1367] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240503 02:11:27.458550 28206 raft_consensus.cc:482] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.458811 28201 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.458739 28206 raft_consensus.cc:504] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.458966 28200 raft_consensus.cc:348] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.459169 28200 raft_consensus.cc:374] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.459061 28199 raft_consensus.cc:348] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.459280 28200 raft_consensus.cc:729] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.459285 28199 raft_consensus.cc:374] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.459470 28199 raft_consensus.cc:729] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.459518 28201 ts_tablet_manager.cc:1367] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:11:27.459558 28200 consensus_queue.cc:260] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.459724 28199 consensus_queue.cc:260] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.459954 28206 leader_election.cc:288] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.460129 28200 ts_tablet_manager.cc:1398] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:27.460445 28200 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: Bootstrap starting.
I20240503 02:11:27.462225 28200 tablet_bootstrap.cc:654] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.463459 28201 raft_consensus.cc:348] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.463820 28201 raft_consensus.cc:374] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.463970 28201 raft_consensus.cc:729] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.464385 28201 consensus_queue.cc:260] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.466712 28201 ts_tablet_manager.cc:1398] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:11:27.468334 28201 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.470584 28201 tablet_bootstrap.cc:654] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.479497 28201 tablet_bootstrap.cc:492] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.479809 28201 ts_tablet_manager.cc:1367] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240503 02:11:27.480921 28201 raft_consensus.cc:348] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.481297 28201 raft_consensus.cc:374] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.481478 28201 raft_consensus.cc:729] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.481884 28201 consensus_queue.cc:260] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.485677 28201 ts_tablet_manager.cc:1398] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:11:27.490139 28201 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: Bootstrap starting.
I20240503 02:11:27.492724 28201 tablet_bootstrap.cc:654] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.502395 28201 tablet_bootstrap.cc:492] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: No bootstrap required, opened a new log
I20240503 02:11:27.502779 28201 ts_tablet_manager.cc:1367] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240503 02:11:27.504001 28201 raft_consensus.cc:348] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.505429 28201 raft_consensus.cc:374] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.505592 28201 raft_consensus.cc:729] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.506028 28201 consensus_queue.cc:260] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.508234 28201 ts_tablet_manager.cc:1398] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:11:27.509369 28205 raft_consensus.cc:482] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.509586 28205 raft_consensus.cc:504] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.510476 28205 leader_election.cc:288] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:27.511237 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
I20240503 02:11:27.511507 28080 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 0.
I20240503 02:11:27.512043 28120 leader_election.cc:302] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, 5b57ee6b7baf4e858fca86cdf3241585; no voters:
I20240503 02:11:27.512451 28205 raft_consensus.cc:2781] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.512606 28205 raft_consensus.cc:482] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.512743 28205 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.514304 28206 raft_consensus.cc:482] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.514508 28206 raft_consensus.cc:504] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.515347 28206 leader_election.cc:288] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.515619 28206 raft_consensus.cc:482] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.515761 28206 raft_consensus.cc:504] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.516489 28206 leader_election.cc:288] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.516767 28205 raft_consensus.cc:504] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.517256 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.517525 28005 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 0.
I20240503 02:11:27.517715 28209 raft_consensus.cc:482] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.517887 28209 raft_consensus.cc:504] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.518195 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.518635 28005 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.519887 28209 leader_election.cc:288] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.522819 28005 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 1.
I20240503 02:11:27.523320 28117 leader_election.cc:302] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.523681 28224 raft_consensus.cc:2781] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.524302 28205 leader_election.cc:288] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:27.527530 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
I20240503 02:11:27.527949 28078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2102533fedbf43ac840cde53fcc060a6" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
I20240503 02:11:27.528122 28079 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 0.
I20240503 02:11:27.527701 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077f639cee044bf863f7c2382180644" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
I20240503 02:11:27.528272 28078 raft_consensus.cc:2454] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 0.
I20240503 02:11:27.528453 28080 raft_consensus.cc:2454] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 0.
I20240503 02:11:27.529206 27971 leader_election.cc:302] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.529240 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077f639cee044bf863f7c2382180644" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.529601 28154 raft_consensus.cc:2454] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 0.
I20240503 02:11:27.529788 27971 leader_election.cc:302] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.530129 28206 raft_consensus.cc:2741] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:11:27.530761 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.531584 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2102533fedbf43ac840cde53fcc060a6" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.531927 28153 raft_consensus.cc:2454] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 0.
I20240503 02:11:27.532444 27968 leader_election.cc:302] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.532784 28206 raft_consensus.cc:2781] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.532907 28206 raft_consensus.cc:482] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.533053 28206 raft_consensus.cc:3037] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.537464 28205 raft_consensus.cc:482] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.537657 28206 raft_consensus.cc:504] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.537750 28205 raft_consensus.cc:504] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.538642 28206 leader_election.cc:288] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.539106 28205 leader_election.cc:288] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:27.539579 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077f639cee044bf863f7c2382180644" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:27.539849 28153 raft_consensus.cc:3037] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.542342 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e077f639cee044bf863f7c2382180644" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:27.542593 28080 raft_consensus.cc:3037] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.544286 28153 raft_consensus.cc:2454] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 1.
I20240503 02:11:27.547706 28080 raft_consensus.cc:2454] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 1.
I20240503 02:11:27.548717 28224 raft_consensus.cc:686] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Becoming Leader. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Running, Role: LEADER
I20240503 02:11:27.549176 28224 consensus_queue.cc:237] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.551647 27968 leader_election.cc:302] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.552690 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c719be39f5684041b2e1e5c116b96a2c" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.553025 28005 raft_consensus.cc:2454] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 0.
I20240503 02:11:27.553471 28117 leader_election.cc:302] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.553853 28236 raft_consensus.cc:2781] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.553942 28206 raft_consensus.cc:2781] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.554131 28236 raft_consensus.cc:482] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.554411 28236 raft_consensus.cc:3037] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.554781 28229 raft_consensus.cc:2781] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.554142 28206 raft_consensus.cc:482] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.555240 28199 ts_tablet_manager.cc:1398] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.097s user 0.031s sys 0.046s
I20240503 02:11:27.555591 28206 raft_consensus.cc:3037] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.556174 28199 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Bootstrap starting.
I20240503 02:11:27.557543 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:27.557837 28080 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.558549 28078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c719be39f5684041b2e1e5c116b96a2c" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585" is_pre_election: true
I20240503 02:11:27.559239 28199 tablet_bootstrap.cc:654] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.560796 28078 raft_consensus.cc:2454] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 0.
I20240503 02:11:27.563354 28200 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: No bootstrap required, opened a new log
I20240503 02:11:27.564455 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff4d1a84e94055a2063802a103eb66" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.564651 28005 raft_consensus.cc:2454] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.566123 28080 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 1.
I20240503 02:11:27.566473 28200 ts_tablet_manager.cc:1367] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: Time spent bootstrapping tablet: real 0.104s user 0.033s sys 0.063s
I20240503 02:11:27.567863 28043 leader_election.cc:302] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.568140 28209 raft_consensus.cc:2781] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.569140 28209 raft_consensus.cc:482] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.569260 28209 raft_consensus.cc:3037] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.568583 27901 catalog_manager.cc:5484] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289c6211929b44bcb758b1a9079fde00 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "289c6211929b44bcb758b1a9079fde00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:27.571223 28200 raft_consensus.cc:348] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.571962 28200 raft_consensus.cc:374] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.572062 28200 raft_consensus.cc:729] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.572903 28200 consensus_queue.cc:260] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.573442 28200 ts_tablet_manager.cc:1398] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:11:27.576848 28209 raft_consensus.cc:504] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.577747 28209 leader_election.cc:288] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.578285 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff4d1a84e94055a2063802a103eb66" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.578547 28005 raft_consensus.cc:3037] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.580034 28206 raft_consensus.cc:504] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.580703 28236 raft_consensus.cc:504] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.581142 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2102533fedbf43ac840cde53fcc060a6" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:27.581352 28080 raft_consensus.cc:3037] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.581808 28209 raft_consensus.cc:482] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.581892 28005 raft_consensus.cc:2454] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 1.
I20240503 02:11:27.582016 28236 leader_election.cc:288] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049)
I20240503 02:11:27.582377 28229 raft_consensus.cc:686] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 LEADER]: Becoming Leader. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Running, Role: LEADER
I20240503 02:11:27.583070 28199 tablet_bootstrap.cc:492] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: No bootstrap required, opened a new log
I20240503 02:11:27.583037 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff4d1a84e94055a2063802a103eb66" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.583154 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff4d1a84e94055a2063802a103eb66" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:27.583305 28154 raft_consensus.cc:2454] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.583678 28199 ts_tablet_manager.cc:1367] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.011s
I20240503 02:11:27.583669 28229 consensus_queue.cc:237] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.584302 28080 raft_consensus.cc:2454] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 1.
I20240503 02:11:27.584553 28199 raft_consensus.cc:348] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.584667 27971 leader_election.cc:302] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.584796 28199 raft_consensus.cc:374] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.584883 28199 raft_consensus.cc:729] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.581745 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c719be39f5684041b2e1e5c116b96a2c" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.585222 28199 consensus_queue.cc:260] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.585273 28209 raft_consensus.cc:504] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.585405 28004 raft_consensus.cc:3037] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.585274 28043 leader_election.cc:302] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters: 289c6211929b44bcb758b1a9079fde00
I20240503 02:11:27.586930 28240 raft_consensus.cc:2781] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.587116 28240 raft_consensus.cc:686] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 LEADER]: Becoming Leader. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Running, Role: LEADER
I20240503 02:11:27.587497 28240 consensus_queue.cc:237] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.589048 28080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c719be39f5684041b2e1e5c116b96a2c" candidate_uuid: "289c6211929b44bcb758b1a9079fde00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:27.589303 28080 raft_consensus.cc:3037] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.590257 28243 raft_consensus.cc:2781] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.590854 28243 raft_consensus.cc:686] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Becoming Leader. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Running, Role: LEADER
I20240503 02:11:27.591493 28243 consensus_queue.cc:237] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.591715 28080 raft_consensus.cc:2454] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 1.
I20240503 02:11:27.592109 28004 raft_consensus.cc:2454] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c6211929b44bcb758b1a9079fde00 in term 1.
I20240503 02:11:27.593690 28120 leader_election.cc:302] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, 5b57ee6b7baf4e858fca86cdf3241585; no voters:
I20240503 02:11:27.593786 28209 leader_election.cc:288] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.593886 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.594017 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94b14b51bad422b9d559b7b7a48855d" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.594321 28004 raft_consensus.cc:2379] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b57ee6b7baf4e858fca86cdf3241585 in current term 1: Already voted for candidate 289c6211929b44bcb758b1a9079fde00 in this term.
I20240503 02:11:27.594354 28236 raft_consensus.cc:2781] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.594553 28236 raft_consensus.cc:686] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Becoming Leader. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Running, Role: LEADER
I20240503 02:11:27.594424 27901 catalog_manager.cc:5484] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "5b57ee6b7baf4e858fca86cdf3241585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.594911 28236 consensus_queue.cc:237] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.596459 28043 leader_election.cc:302] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585; no voters: 289c6211929b44bcb758b1a9079fde00, e0cf2cd06e8f41e8b1f28c0adcda6fd2
I20240503 02:11:27.597023 28209 raft_consensus.cc:3037] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.598495 28199 ts_tablet_manager.cc:1398] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:11:27.598935 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2102533fedbf43ac840cde53fcc060a6" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:27.599112 28154 raft_consensus.cc:3037] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.599409 27901 catalog_manager.cc:5484] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.600103 27903 catalog_manager.cc:5484] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289c6211929b44bcb758b1a9079fde00 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "289c6211929b44bcb758b1a9079fde00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:27.601652 28154 raft_consensus.cc:2454] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 1.
I20240503 02:11:27.604907 28209 raft_consensus.cc:2726] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:27.605065 28206 leader_election.cc:288] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.606992 27903 catalog_manager.cc:5484] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.614455 28209 raft_consensus.cc:482] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.614645 28209 raft_consensus.cc:504] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.615248 28209 leader_election.cc:288] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.615775 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5b6c37226649e0865bde82646c1ce7" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.615859 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5b6c37226649e0865bde82646c1ce7" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.615975 28004 raft_consensus.cc:2454] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.616111 28154 raft_consensus.cc:2454] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.616379 28043 leader_election.cc:302] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.616640 28209 raft_consensus.cc:2781] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.616724 28209 raft_consensus.cc:482] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.616827 28209 raft_consensus.cc:3037] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.618734 28209 raft_consensus.cc:504] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.619441 28209 leader_election.cc:288] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.619727 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5b6c37226649e0865bde82646c1ce7" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.619872 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e5b6c37226649e0865bde82646c1ce7" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:27.619961 28004 raft_consensus.cc:3037] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.620093 28154 raft_consensus.cc:3037] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.622076 28004 raft_consensus.cc:2454] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 1.
I20240503 02:11:27.622176 28154 raft_consensus.cc:2454] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 1.
I20240503 02:11:27.622498 28043 leader_election.cc:302] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.622768 28209 raft_consensus.cc:2781] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.622936 28209 raft_consensus.cc:686] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Becoming Leader. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Running, Role: LEADER
I20240503 02:11:27.623190 28209 consensus_queue.cc:237] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.625777 27903 catalog_manager.cc:5484] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "5b57ee6b7baf4e858fca86cdf3241585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.653193 28209 raft_consensus.cc:482] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.653403 28209 raft_consensus.cc:504] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.654146 28209 leader_election.cc:288] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.654774 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f343ae46274a9da3ea761fbc614d1e" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00" is_pre_election: true
I20240503 02:11:27.654839 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f343ae46274a9da3ea761fbc614d1e" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" is_pre_election: true
I20240503 02:11:27.654978 28154 raft_consensus.cc:2454] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.655056 28004 raft_consensus.cc:2454] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 0.
I20240503 02:11:27.655402 28043 leader_election.cc:302] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289c6211929b44bcb758b1a9079fde00, 5b57ee6b7baf4e858fca86cdf3241585; no voters:
I20240503 02:11:27.655694 28209 raft_consensus.cc:2781] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.655800 28209 raft_consensus.cc:482] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.655889 28209 raft_consensus.cc:3037] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.658012 28209 raft_consensus.cc:504] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.658532 28209 leader_election.cc:288] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Requested vote from peers e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65:35101), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:27.658986 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f343ae46274a9da3ea761fbc614d1e" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
I20240503 02:11:27.659037 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f343ae46274a9da3ea761fbc614d1e" candidate_uuid: "5b57ee6b7baf4e858fca86cdf3241585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:27.659181 28004 raft_consensus.cc:3037] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.659247 28154 raft_consensus.cc:3037] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.661201 28004 raft_consensus.cc:2454] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 1.
I20240503 02:11:27.661242 28154 raft_consensus.cc:2454] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57ee6b7baf4e858fca86cdf3241585 in term 1.
I20240503 02:11:27.661572 28043 leader_election.cc:302] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:27.661823 28209 raft_consensus.cc:2781] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.661971 28209 raft_consensus.cc:686] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Becoming Leader. State: Replica: 5b57ee6b7baf4e858fca86cdf3241585, State: Running, Role: LEADER
I20240503 02:11:27.662233 28209 consensus_queue.cc:237] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:27.664870 27903 catalog_manager.cc:5484] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "5b57ee6b7baf4e858fca86cdf3241585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.709060 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:27.712476 28251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.712776 28252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.713663 28254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:27.714004 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:27.714242 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:27.714313 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:27.714354 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702287714340 us; error 0 us; skew 500 ppm
I20240503 02:11:27.714493 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:27.715821 27337 webserver.cc:460] Webserver started at http://127.26.178.68:33275/ using document root <none> and password file <none>
I20240503 02:11:27.716040 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:27.716109 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:27.716220 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:27.716837 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-3-root/instance:
uuid: "1bdae48a089d46b58a5493138fe4b4fe"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-0cbc"
I20240503 02:11:27.719429 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:27.721079 28259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:27.721452 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:27.721555 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-3-root
uuid: "1bdae48a089d46b58a5493138fe4b4fe"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-0cbc"
I20240503 02:11:27.721688 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714702282772525-27337-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:27.738330 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:27.739004 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:27.739554 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:27.740413 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:27.740499 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:27.740598 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:27.740681 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:27.758622 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.68:34893
I20240503 02:11:27.758695 28321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.68:34893 every 8 connection(s)
I20240503 02:11:27.766050 28322 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45195
I20240503 02:11:27.766271 28322 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:27.766891 28322 heartbeater.cc:511] Master 127.26.178.126:45195 requested a full tablet report, sending...
I20240503 02:11:27.767824 27903 ts_manager.cc:194] Registered new tserver with Master: 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68:34893)
I20240503 02:11:27.768797 27903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34020
I20240503 02:11:27.878242 28236 consensus_queue.cc:1035] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:27.883090 28205 consensus_queue.cc:1035] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:27.978358 27956 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:27.987093 28154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f"
dest_uuid: "289c6211929b44bcb758b1a9079fde00"
mode: GRACEFUL
new_leader_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
from {username='slave'} at 127.0.0.1:39042
I20240503 02:11:27.987484 28154 raft_consensus.cc:595] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Received request to transfer leadership to TS e0cf2cd06e8f41e8b1f28c0adcda6fd2
I20240503 02:11:27.988011 27956 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 02:11:27.988323 27956 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:28.005806 28209 consensus_queue.cc:1035] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.010371 28209 consensus_queue.cc:1035] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.014983 28209 consensus_queue.cc:1035] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.017084 28205 consensus_queue.cc:1035] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.018200 28154 raft_consensus.cc:3037] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.020630 28240 consensus_queue.cc:1035] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.022353 28236 raft_consensus.cc:979] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: : Instructing follower e0cf2cd06e8f41e8b1f28c0adcda6fd2 to start an election
I20240503 02:11:28.023144 28236 raft_consensus.cc:1067] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Signalling peer e0cf2cd06e8f41e8b1f28c0adcda6fd2 to start an election
I20240503 02:11:28.025290 28005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f"
dest_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2"
from {username='slave'} at 127.0.0.1:37688
I20240503 02:11:28.025612 28005 raft_consensus.cc:482] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:28.025743 28005 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:28.029035 28243 consensus_queue.cc:1035] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.030931 28236 consensus_queue.cc:1035] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.032061 28338 consensus_queue.cc:1035] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.034247 28236 consensus_queue.cc:1035] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.037714 28005 raft_consensus.cc:504] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:28.039239 28078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5b57ee6b7baf4e858fca86cdf3241585"
I20240503 02:11:28.039449 28078 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:28.039965 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bd947bec044d7d8ded1d301b33d12f" candidate_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "289c6211929b44bcb758b1a9079fde00"
I20240503 02:11:28.040199 28154 raft_consensus.cc:3032] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:28.040343 28154 raft_consensus.cc:729] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 289c6211929b44bcb758b1a9079fde00, State: Running, Role: LEADER
I20240503 02:11:28.040793 28154 consensus_queue.cc:260] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:28.042312 28154 raft_consensus.cc:3037] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:28.044152 28005 leader_election.cc:288] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 2 election: Requested vote from peers 5b57ee6b7baf4e858fca86cdf3241585 (127.26.178.66:42049), 289c6211929b44bcb758b1a9079fde00 (127.26.178.67:40449)
I20240503 02:11:28.050560 28078 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 2.
I20240503 02:11:28.050592 28248 consensus_queue.cc:1035] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:11:28.050791 28154 raft_consensus.cc:2454] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cf2cd06e8f41e8b1f28c0adcda6fd2 in term 2.
I20240503 02:11:28.050988 27971 leader_election.cc:302] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57ee6b7baf4e858fca86cdf3241585, e0cf2cd06e8f41e8b1f28c0adcda6fd2; no voters:
I20240503 02:11:28.051273 28248 raft_consensus.cc:2781] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:28.051442 28248 raft_consensus.cc:686] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 2 LEADER]: Becoming Leader. State: Replica: e0cf2cd06e8f41e8b1f28c0adcda6fd2, State: Running, Role: LEADER
I20240503 02:11:28.051754 28248 consensus_queue.cc:237] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } }
I20240503 02:11:28.054329 27901 catalog_manager.cc:5484] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 reported cstate change: term changed from 1 to 2, leader changed from 289c6211929b44bcb758b1a9079fde00 (127.26.178.67) to e0cf2cd06e8f41e8b1f28c0adcda6fd2 (127.26.178.65). New cstate: current_term: 2 leader_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:28.095584 28243 consensus_queue.cc:1035] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.099661 28335 consensus_queue.cc:1035] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.103008 28248 consensus_queue.cc:1035] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.106925 28248 consensus_queue.cc:1035] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.626785 28153 raft_consensus.cc:1261] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 2 FOLLOWER]: Refusing update from remote peer e0cf2cd06e8f41e8b1f28c0adcda6fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:11:28.627367 28241 consensus_queue.cc:1035] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:28.631042 28078 raft_consensus.cc:1261] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 2 FOLLOWER]: Refusing update from remote peer e0cf2cd06e8f41e8b1f28c0adcda6fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:11:28.631567 28241 consensus_queue.cc:1035] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:28.770001 28322 heartbeater.cc:503] Master 127.26.178.126:45195 was elected leader, sending a full tablet report...
I20240503 02:11:29.988720 27956 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:29.990149 27956 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:29.990324 27956 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:31.990738 27956 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:31.992202 27956 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:31.992393 27956 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:32.989351 28153 consensus_queue.cc:237] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } }
I20240503 02:11:32.992569 28078 raft_consensus.cc:1261] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Refusing update from remote peer 289c6211929b44bcb758b1a9079fde00: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:32.992730 28005 raft_consensus.cc:1261] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Refusing update from remote peer 289c6211929b44bcb758b1a9079fde00: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:32.993222 28352 consensus_queue.cc:1035] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:32.993608 28357 consensus_queue.cc:1035] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:32.996753 28357 raft_consensus.cc:2932] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:32.997530 28078 raft_consensus.cc:2932] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:32.997669 28005 raft_consensus.cc:2932] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
W20240503 02:11:33.003705 28118 consensus_peers.cc:485] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 -> Peer 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68:34893): Couldn't send request to peer 1bdae48a089d46b58a5493138fe4b4fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c719be39f5684041b2e1e5c116b96a2c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.003895 27903 catalog_manager.cc:5484] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "289c6211929b44bcb758b1a9079fde00" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:33.012413 28078 consensus_queue.cc:237] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } }
I20240503 02:11:33.014863 28005 raft_consensus.cc:1261] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Refusing update from remote peer 5b57ee6b7baf4e858fca86cdf3241585: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.015431 28368 consensus_queue.cc:1035] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.015633 28153 raft_consensus.cc:1261] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Refusing update from remote peer 5b57ee6b7baf4e858fca86cdf3241585: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.016141 28368 consensus_queue.cc:1035] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.018440 28245 raft_consensus.cc:2932] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.022462 28005 raft_consensus.cc:2932] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.023757 28078 consensus_queue.cc:237] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } }
W20240503 02:11:33.027599 28044 consensus_peers.cc:485] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 -> Peer 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68:34893): Couldn't send request to peer 1bdae48a089d46b58a5493138fe4b4fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eff4d1a84e94055a2063802a103eb66. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:33.028746 28044 consensus_peers.cc:485] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 -> Peer 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68:34893): Couldn't send request to peer 1bdae48a089d46b58a5493138fe4b4fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91f343ae46274a9da3ea761fbc614d1e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.029676 28003 raft_consensus.cc:1261] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Refusing update from remote peer 5b57ee6b7baf4e858fca86cdf3241585: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.029528 27902 catalog_manager.cc:5484] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "5b57ee6b7baf4e858fca86cdf3241585" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:33.030421 28368 consensus_queue.cc:1035] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.038008 28154 raft_consensus.cc:1261] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Refusing update from remote peer 5b57ee6b7baf4e858fca86cdf3241585: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.038597 28368 consensus_queue.cc:1035] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.039028 28154 raft_consensus.cc:2932] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.043797 28245 raft_consensus.cc:2932] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.044428 28005 raft_consensus.cc:2932] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.047021 28153 raft_consensus.cc:2932] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.049393 27901 catalog_manager.cc:5484] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "5b57ee6b7baf4e858fca86cdf3241585" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:33.056746 28005 consensus_queue.cc:237] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } }
I20240503 02:11:33.059435 28078 raft_consensus.cc:1261] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Refusing update from remote peer e0cf2cd06e8f41e8b1f28c0adcda6fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.059497 28153 raft_consensus.cc:1261] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Refusing update from remote peer e0cf2cd06e8f41e8b1f28c0adcda6fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.059998 28241 consensus_queue.cc:1035] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.060427 28379 consensus_queue.cc:1035] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:33.064052 28378 raft_consensus.cc:2932] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.064770 28078 raft_consensus.cc:2932] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.064896 28153 raft_consensus.cc:2932] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
W20240503 02:11:33.070017 27969 consensus_peers.cc:485] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 -> Peer 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68:34893): Couldn't send request to peer 1bdae48a089d46b58a5493138fe4b4fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2102533fedbf43ac840cde53fcc060a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.070427 27903 catalog_manager.cc:5484] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 reported cstate change: config changed from index -1 to 2, NON_VOTER 1bdae48a089d46b58a5493138fe4b4fe (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e0cf2cd06e8f41e8b1f28c0adcda6fd2" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 35101 } } peers { permanent_uuid: "5b57ee6b7baf4e858fca86cdf3241585" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 42049 } } peers { permanent_uuid: "289c6211929b44bcb758b1a9079fde00" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 40449 } attrs { replace: true } } peers { permanent_uuid: "1bdae48a089d46b58a5493138fe4b4fe" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 34893 } attrs { promote: true } } }
I20240503 02:11:33.274221 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
W20240503 02:11:33.280699 27887 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240503 02:11:33.283913 27951 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:11:33.284096 27951 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:11:33.284919 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:33.285300 27337 tablet_replica.cc:330] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.285529 27337 raft_consensus.cc:2229] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.285815 27337 raft_consensus.cc:2258] T 91f343ae46274a9da3ea761fbc614d1e P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.286640 27337 tablet_replica.cc:330] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.286906 27337 raft_consensus.cc:2229] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.287226 27337 raft_consensus.cc:2258] T 31bd947bec044d7d8ded1d301b33d12f P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.287767 27337 tablet_replica.cc:330] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.287935 27337 raft_consensus.cc:2229] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.288122 27337 raft_consensus.cc:2258] T 6eff4d1a84e94055a2063802a103eb66 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.288734 27337 tablet_replica.cc:330] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.288910 27337 raft_consensus.cc:2229] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.289142 27337 raft_consensus.cc:2258] T e077f639cee044bf863f7c2382180644 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.289572 27337 tablet_replica.cc:330] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.289713 27337 raft_consensus.cc:2229] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.289891 27337 raft_consensus.cc:2258] T c719be39f5684041b2e1e5c116b96a2c P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.290519 27337 tablet_replica.cc:330] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.290740 27337 raft_consensus.cc:2229] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.290894 27337 raft_consensus.cc:2258] T d94b14b51bad422b9d559b7b7a48855d P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.291467 27337 tablet_replica.cc:330] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.291640 27337 raft_consensus.cc:2229] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.292021 27337 raft_consensus.cc:2258] T 2102533fedbf43ac840cde53fcc060a6 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.292721 27337 tablet_replica.cc:330] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2: stopping tablet replica
I20240503 02:11:33.292891 27337 raft_consensus.cc:2229] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.293051 27337 raft_consensus.cc:2258] T 6e5b6c37226649e0865bde82646c1ce7 P e0cf2cd06e8f41e8b1f28c0adcda6fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.299634 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:11:33.304728 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
W20240503 02:11:33.307524 27951 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:11:33.308194 27951 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:11:33.314837 27951 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:11:33.314944 27951 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:11:33.315526 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:33.315979 27337 tablet_replica.cc:330] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.316314 27337 raft_consensus.cc:2229] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.316669 27337 raft_consensus.cc:2258] T 6e5b6c37226649e0865bde82646c1ce7 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.317346 27337 tablet_replica.cc:330] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.317520 27337 raft_consensus.cc:2229] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.317759 27337 raft_consensus.cc:2258] T 2102533fedbf43ac840cde53fcc060a6 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.318763 27337 tablet_replica.cc:330] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.318961 27337 raft_consensus.cc:2229] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.319341 27337 raft_consensus.cc:2258] T 6eff4d1a84e94055a2063802a103eb66 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.320060 27337 tablet_replica.cc:330] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.320237 27337 raft_consensus.cc:2229] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.320420 27337 raft_consensus.cc:2258] T e077f639cee044bf863f7c2382180644 P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.320866 27337 tablet_replica.cc:330] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.321033 27337 raft_consensus.cc:2229] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.321184 27337 raft_consensus.cc:2258] T 31bd947bec044d7d8ded1d301b33d12f P 5b57ee6b7baf4e858fca86cdf3241585 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.321630 27337 tablet_replica.cc:330] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.321784 27337 raft_consensus.cc:2229] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.322118 27337 raft_consensus.cc:2258] T 91f343ae46274a9da3ea761fbc614d1e P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.322829 27337 tablet_replica.cc:330] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.322999 27337 raft_consensus.cc:2229] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.323186 27337 raft_consensus.cc:2258] T c719be39f5684041b2e1e5c116b96a2c P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.324013 27337 tablet_replica.cc:330] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585: stopping tablet replica
I20240503 02:11:33.324162 27337 raft_consensus.cc:2229] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.324340 27337 raft_consensus.cc:2258] T d94b14b51bad422b9d559b7b7a48855d P 5b57ee6b7baf4e858fca86cdf3241585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.340751 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:11:33.345513 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
W20240503 02:11:33.348757 27951 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:11:33.348919 27951 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:11:33.354380 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:33.354739 27337 tablet_replica.cc:330] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.354936 27337 raft_consensus.cc:2229] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.355144 27337 raft_consensus.cc:2258] T 91f343ae46274a9da3ea761fbc614d1e P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.355944 27337 tablet_replica.cc:330] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.356124 27337 raft_consensus.cc:2229] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.356297 27337 raft_consensus.cc:2258] T 31bd947bec044d7d8ded1d301b33d12f P 289c6211929b44bcb758b1a9079fde00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.356781 27337 tablet_replica.cc:330] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.356914 27337 raft_consensus.cc:2229] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.357095 27337 raft_consensus.cc:2258] T 6eff4d1a84e94055a2063802a103eb66 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.357757 27337 tablet_replica.cc:330] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.357915 27337 raft_consensus.cc:2229] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.358036 27337 raft_consensus.cc:2258] T e077f639cee044bf863f7c2382180644 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.358417 27337 tablet_replica.cc:330] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.358528 27337 raft_consensus.cc:2229] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.358709 27337 raft_consensus.cc:2258] T 2102533fedbf43ac840cde53fcc060a6 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.359264 27337 tablet_replica.cc:330] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.359383 27337 raft_consensus.cc:2229] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:33.359493 27337 raft_consensus.cc:2258] T 6e5b6c37226649e0865bde82646c1ce7 P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.359848 27337 tablet_replica.cc:330] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.359947 27337 raft_consensus.cc:2229] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.360198 27337 raft_consensus.cc:2258] T d94b14b51bad422b9d559b7b7a48855d P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.360571 27337 tablet_replica.cc:330] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00: stopping tablet replica
I20240503 02:11:33.360683 27337 raft_consensus.cc:2229] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.360941 27337 raft_consensus.cc:2258] T c719be39f5684041b2e1e5c116b96a2c P 289c6211929b44bcb758b1a9079fde00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.376648 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:11:33.380868 27337 tablet_server.cc:178] TabletServer@127.26.178.68:0 shutting down...
I20240503 02:11:33.389452 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:33.402866 27337 tablet_server.cc:194] TabletServer@127.26.178.68:0 shutdown complete.
I20240503 02:11:33.405668 27337 master.cc:537] Master@127.26.178.126:45195 shutting down...
I20240503 02:11:33.412636 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:33.412904 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:33.413010 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 517961dab66c4502be552554838b650e: stopping tablet replica
I20240503 02:11:33.416977 27337 master.cc:552] Master@127.26.178.126:45195 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7515 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 02:11:33.434526 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:37647,127.26.178.125:41695,127.26.178.124:43637
I20240503 02:11:33.435254 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:33.438102 28384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.438442 28385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.439692 28387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.440730 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:33.441035 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.441107 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.441184 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293441149 us; error 0 us; skew 500 ppm
I20240503 02:11:33.441346 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.442772 27337 webserver.cc:460] Webserver started at http://127.26.178.126:46857/ using document root <none> and password file <none>
I20240503 02:11:33.443015 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.443126 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.443277 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.443956 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.446576 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:11:33.448383 28392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.448813 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:33.448911 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.449043 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.458323 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.459017 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.471520 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:37647
I20240503 02:11:33.471568 28443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:37647 every 8 connection(s)
I20240503 02:11:33.472906 28444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.472978 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:33.475849 28446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.476061 28447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.477069 28449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.477113 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.477891 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:33.478402 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.478500 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.478585 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293478539 us; error 0 us; skew 500 ppm
I20240503 02:11:33.478796 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.480409 27337 webserver.cc:460] Webserver started at http://127.26.178.125:46241/ using document root <none> and password file <none>
I20240503 02:11:33.480684 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.480801 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.480950 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.481796 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root/instance:
uuid: "c9dfd4f415274063b8c239ba4468e31c"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.484880 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:33.484851 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.486842 28457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20240503 02:11:33.487066 28444 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.26.178.125:41695: Network error: Client connection negotiation failed: client connection to 127.26.178.125:41695: connect: Connection refused (error 111)
I20240503 02:11:33.487269 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:33.487424 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
uuid: "c9dfd4f415274063b8c239ba4468e31c"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.487612 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.497377 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.498032 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.508330 28444 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } attempt: 1
W20240503 02:11:33.510478 28444 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.26.178.125:41695: Network error: Client connection negotiation failed: client connection to 127.26.178.125:41695: connect: Connection refused (error 111)
I20240503 02:11:33.510562 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.125:41695
I20240503 02:11:33.510620 28508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.125:41695 every 8 connection(s)
I20240503 02:11:33.511914 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.512020 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:33.515389 28511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.515636 28509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:11:33.516525 28514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.516960 27337 server_base.cc:1034] running on GCE node
W20240503 02:11:33.517355 28512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.517784 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.517860 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.517937 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293517899 us; error 0 us; skew 500 ppm
I20240503 02:11:33.518148 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.519680 27337 webserver.cc:460] Webserver started at http://127.26.178.124:44047/ using document root <none> and password file <none>
I20240503 02:11:33.519907 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.520010 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.520160 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.521031 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root/instance:
uuid: "2d4c13ce504b4085b63ee5cff200519a"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.522385 28509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.524358 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:33.526141 28521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.526521 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:33.526625 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
uuid: "2d4c13ce504b4085b63ee5cff200519a"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.526793 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.529306 28509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:11:33.531257 28509 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.26.178.124:43637: Network error: Client connection negotiation failed: client connection to 127.26.178.124:43637: connect: Connection refused (error 111)
I20240503 02:11:33.550580 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.551290 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.558755 28444 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } attempt: 2
I20240503 02:11:33.565032 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.124:43637
I20240503 02:11:33.565094 28573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.124:43637 every 8 connection(s)
I20240503 02:11:33.565624 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.566133 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:11:33.566713 28574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.570190 28574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.575129 28444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: Bootstrap starting.
I20240503 02:11:33.576467 28574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.576718 28444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.578812 28444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: No bootstrap required, opened a new log
I20240503 02:11:33.579679 28444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.579927 28444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.580015 28444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0f3c7fc0564b9f9af72ded16fd60ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.580255 28444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.580907 28579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.581171 28444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:33.581262 28579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.583190 28574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:11:33.584458 28509 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } attempt: 1
I20240503 02:11:33.592926 28574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: Bootstrap starting.
I20240503 02:11:33.594336 28574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.596458 28574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: No bootstrap required, opened a new log
I20240503 02:11:33.596635 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 02:11:33.597127 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: Bootstrap starting.
I20240503 02:11:33.597321 28574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.597544 28574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.597649 28574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d4c13ce504b4085b63ee5cff200519a, State: Initialized, Role: FOLLOWER
W20240503 02:11:33.597978 28592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:33.598060 28592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:11:33.597942 28574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.598497 28509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.598644 28593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.598896 28593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.599004 28574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:33.600440 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: No bootstrap required, opened a new log
I20240503 02:11:33.601056 28509 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.601239 28509 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.601336 28509 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9dfd4f415274063b8c239ba4468e31c, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.601624 28509 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.602237 28600 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.602424 28509 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:33.602537 28600 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: This master's current role is: FOLLOWER
W20240503 02:11:33.605726 28609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:33.605866 28609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:11:33.607113 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 02:11:33.607843 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:33.607990 28617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:33.608078 28617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:11:33.611341 28618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.611361 28619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.612413 28621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.613394 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:33.613663 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.613770 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.613835 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293613817 us; error 0 us; skew 500 ppm
I20240503 02:11:33.614012 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.615401 27337 webserver.cc:460] Webserver started at http://127.26.178.65:46881/ using document root <none> and password file <none>
I20240503 02:11:33.615654 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.615741 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.615895 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.616583 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "f63d7e34bd1a4488a2dd7e9ac3216697"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.619131 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:11:33.620774 28626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.621105 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:11:33.621246 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "f63d7e34bd1a4488a2dd7e9ac3216697"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.621373 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.636023 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.636644 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.637255 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:33.638170 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:33.638267 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.638370 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:33.638443 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.653447 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:37155
I20240503 02:11:33.653493 28688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:37155 every 8 connection(s)
I20240503 02:11:33.662472 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:33.672188 28579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:33.672472 28579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.673584 28579 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695), 2d4c13ce504b4085b63ee5cff200519a (127.26.178.124:43637)
I20240503 02:11:33.674433 28549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d4c13ce504b4085b63ee5cff200519a" is_pre_election: true
I20240503 02:11:33.674772 28549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0f3c7fc0564b9f9af72ded16fd60ae in term 0.
I20240503 02:11:33.675051 28484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c" is_pre_election: true
I20240503 02:11:33.675390 28395 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a, 6c0f3c7fc0564b9f9af72ded16fd60ae; no voters:
I20240503 02:11:33.675390 28484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0f3c7fc0564b9f9af72ded16fd60ae in term 0.
I20240503 02:11:33.676640 28579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:33.676802 28579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:33.676910 28579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:11:33.677680 28696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.676662 28689 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37647
W20240503 02:11:33.678326 28697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.678503 28689 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.678704 28691 heartbeater.cc:347] Connected to a master server at 127.26.178.125:41695
I20240503 02:11:33.678828 28691 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.679101 28689 heartbeater.cc:511] Master 127.26.178.126:37647 requested a full tablet report, sending...
I20240503 02:11:33.679255 28691 heartbeater.cc:511] Master 127.26.178.125:41695 requested a full tablet report, sending...
I20240503 02:11:33.680233 28474 ts_manager.cc:194] Registered new tserver with Master: f63d7e34bd1a4488a2dd7e9ac3216697 (127.26.178.65:37155)
I20240503 02:11:33.681061 28690 heartbeater.cc:347] Connected to a master server at 127.26.178.124:43637
I20240503 02:11:33.681228 28690 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.681668 28690 heartbeater.cc:511] Master 127.26.178.124:43637 requested a full tablet report, sending...
I20240503 02:11:33.682919 28539 ts_manager.cc:194] Registered new tserver with Master: f63d7e34bd1a4488a2dd7e9ac3216697 (127.26.178.65:37155)
I20240503 02:11:33.682963 28579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
W20240503 02:11:33.683123 28699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.683732 28579 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [CANDIDATE]: Term 1 election: Requested vote from peers c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695), 2d4c13ce504b4085b63ee5cff200519a (127.26.178.124:43637)
I20240503 02:11:33.683773 28409 ts_manager.cc:194] Registered new tserver with Master: f63d7e34bd1a4488a2dd7e9ac3216697 (127.26.178.65:37155)
I20240503 02:11:33.684080 28484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c"
I20240503 02:11:33.684198 28549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d4c13ce504b4085b63ee5cff200519a"
I20240503 02:11:33.684288 28484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:33.684391 28549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:33.684455 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:33.684791 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.684881 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.684962 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293684921 us; error 0 us; skew 500 ppm
I20240503 02:11:33.685146 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.686702 28484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0f3c7fc0564b9f9af72ded16fd60ae in term 1.
I20240503 02:11:33.687165 28393 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0f3c7fc0564b9f9af72ded16fd60ae, c9dfd4f415274063b8c239ba4468e31c; no voters:
I20240503 02:11:33.687458 28579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:33.687537 28549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0f3c7fc0564b9f9af72ded16fd60ae in term 1.
I20240503 02:11:33.687736 27337 webserver.cc:460] Webserver started at http://127.26.178.66:38239/ using document root <none> and password file <none>
I20240503 02:11:33.687889 28579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 1 LEADER]: Becoming Leader. State: Replica: 6c0f3c7fc0564b9f9af72ded16fd60ae, State: Running, Role: LEADER
I20240503 02:11:33.688042 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.688156 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.688297 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.688227 28579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:33.688985 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "bc696ce923844cc1887f8eaa779020ad"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.689491 28703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0f3c7fc0564b9f9af72ded16fd60ae. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.689771 28703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:33.690328 28705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:33.691857 28705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:33.692127 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:11:33.694106 28708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.694742 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:33.694878 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "bc696ce923844cc1887f8eaa779020ad"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.695044 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.695259 28484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Refusing update from remote peer 6c0f3c7fc0564b9f9af72ded16fd60ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.695261 28549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Refusing update from remote peer 6c0f3c7fc0564b9f9af72ded16fd60ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:33.695858 28579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.696115 28703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.707340 28600 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0f3c7fc0564b9f9af72ded16fd60ae. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.707650 28600 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.711623 28593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0f3c7fc0564b9f9af72ded16fd60ae. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.711915 28593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.713294 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.714053 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.714640 28593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.714973 28593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.716204 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:33.717182 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:33.717265 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.717409 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:33.717478 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.719466 28600 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.719766 28600 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:33.721240 28705 catalog_manager.cc:1347] Generated new cluster ID: cb86c775dde94af0b361114c71bd3de8
I20240503 02:11:33.721369 28705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:33.727456 28703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.727656 28703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:33.727072 28579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:33.727811 28579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:33.744153 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:34247
I20240503 02:11:33.744180 28779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:34247 every 8 connection(s)
I20240503 02:11:33.751566 28705 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:33.752667 28705 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:33.764873 28780 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37647
I20240503 02:11:33.765061 28780 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.766566 28781 heartbeater.cc:347] Connected to a master server at 127.26.178.124:43637
I20240503 02:11:33.766754 28781 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.767328 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:33.768524 28780 heartbeater.cc:511] Master 127.26.178.126:37647 requested a full tablet report, sending...
I20240503 02:11:33.769759 28409 ts_manager.cc:194] Registered new tserver with Master: bc696ce923844cc1887f8eaa779020ad (127.26.178.66:34247)
I20240503 02:11:33.770712 28781 heartbeater.cc:511] Master 127.26.178.124:43637 requested a full tablet report, sending...
I20240503 02:11:33.771669 28705 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: Generated new TSK 0
I20240503 02:11:33.772064 28705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:33.773592 28539 ts_manager.cc:194] Registered new tserver with Master: bc696ce923844cc1887f8eaa779020ad (127.26.178.66:34247)
W20240503 02:11:33.779201 28786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.779666 28787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.780867 28789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.781469 28782 heartbeater.cc:347] Connected to a master server at 127.26.178.125:41695
I20240503 02:11:33.781615 28782 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.781997 28782 heartbeater.cc:511] Master 127.26.178.125:41695 requested a full tablet report, sending...
I20240503 02:11:33.782140 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:33.782435 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:33.782537 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:33.782594 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702293782571 us; error 0 us; skew 500 ppm
I20240503 02:11:33.782810 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.783110 28474 ts_manager.cc:194] Registered new tserver with Master: bc696ce923844cc1887f8eaa779020ad (127.26.178.66:34247)
I20240503 02:11:33.784169 27337 webserver.cc:460] Webserver started at http://127.26.178.67:35019/ using document root <none> and password file <none>
I20240503 02:11:33.784391 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.784489 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.784610 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.785207 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.787659 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:33.789268 28794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.789600 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:33.789719 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-0cbc"
I20240503 02:11:33.789824 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.802651 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.803319 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.803941 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:33.804822 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:33.804903 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.805009 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:33.805074 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.820760 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:44243
I20240503 02:11:33.820951 28857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:44243 every 8 connection(s)
I20240503 02:11:33.833595 28858 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37647
I20240503 02:11:33.833832 28858 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.834575 28858 heartbeater.cc:511] Master 127.26.178.126:37647 requested a full tablet report, sending...
I20240503 02:11:33.836022 28409 ts_manager.cc:194] Registered new tserver with Master: ca9793eaecbd4479b47b6b7c4fb22d8f (127.26.178.67:44243)
I20240503 02:11:33.838176 28409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43780
I20240503 02:11:33.839473 28859 heartbeater.cc:347] Connected to a master server at 127.26.178.124:43637
I20240503 02:11:33.839650 28859 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.839942 28859 heartbeater.cc:511] Master 127.26.178.124:43637 requested a full tablet report, sending...
I20240503 02:11:33.840914 28539 ts_manager.cc:194] Registered new tserver with Master: ca9793eaecbd4479b47b6b7c4fb22d8f (127.26.178.67:44243)
I20240503 02:11:33.843441 28860 heartbeater.cc:347] Connected to a master server at 127.26.178.125:41695
I20240503 02:11:33.843595 28860 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.843881 28860 heartbeater.cc:511] Master 127.26.178.125:41695 requested a full tablet report, sending...
I20240503 02:11:33.844815 28474 ts_manager.cc:194] Registered new tserver with Master: ca9793eaecbd4479b47b6b7c4fb22d8f (127.26.178.67:44243)
I20240503 02:11:33.845152 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015196609s
I20240503 02:11:34.607234 28609 catalog_manager.cc:1259] Loaded cluster ID: cb86c775dde94af0b361114c71bd3de8
I20240503 02:11:34.607353 28609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: loading cluster ID for follower catalog manager: success
I20240503 02:11:34.609292 28609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: acquiring CA information for follower catalog manager: success
I20240503 02:11:34.609555 28617 catalog_manager.cc:1259] Loaded cluster ID: cb86c775dde94af0b361114c71bd3de8
I20240503 02:11:34.609642 28617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: loading cluster ID for follower catalog manager: success
I20240503 02:11:34.610438 28609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:11:34.611667 28617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: acquiring CA information for follower catalog manager: success
I20240503 02:11:34.612661 28617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:11:34.686025 28409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43768
I20240503 02:11:34.773077 28409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43778
I20240503 02:11:34.840013 28858 heartbeater.cc:503] Master 127.26.178.126:37647 was elected leader, sending a full tablet report...
I20240503 02:11:35.596794 28591 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:35.687186 28689 heartbeater.cc:503] Master 127.26.178.126:37647 was elected leader, sending a full tablet report...
I20240503 02:11:35.774603 28780 heartbeater.cc:503] Master 127.26.178.126:37647 was elected leader, sending a full tablet report...
I20240503 02:11:35.901069 27337 test_util.cc:268] Using random seed: 1027482468
I20240503 02:11:35.927369 28409 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43788:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:11:35.953431 28744 tablet_service.cc:1467] Processing CreateTablet for tablet 67cee140a3e8458f8294211ac093d121 (DEFAULT_TABLE table=test-workload [id=5ac8c56ed329415fb3101543787153e0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:11:35.953940 28744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cee140a3e8458f8294211ac093d121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.961762 28893 tablet_bootstrap.cc:492] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: Bootstrap starting.
I20240503 02:11:35.964198 28893 tablet_bootstrap.cc:654] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.965253 28822 tablet_service.cc:1467] Processing CreateTablet for tablet 9eeaf3e2329d4e91a5fe215600d0245f (DEFAULT_TABLE table=test-workload [id=5ac8c56ed329415fb3101543787153e0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:11:35.965801 28822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eeaf3e2329d4e91a5fe215600d0245f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.967092 28893 tablet_bootstrap.cc:492] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: No bootstrap required, opened a new log
I20240503 02:11:35.967286 28893 ts_tablet_manager.cc:1367] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:35.968158 28893 raft_consensus.cc:348] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc696ce923844cc1887f8eaa779020ad" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 34247 } }
I20240503 02:11:35.968376 28893 raft_consensus.cc:374] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.968520 28893 raft_consensus.cc:729] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc696ce923844cc1887f8eaa779020ad, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.968462 28654 tablet_service.cc:1467] Processing CreateTablet for tablet 80496c7c7d4f4cd69614016a33272eb4 (DEFAULT_TABLE table=test-workload [id=5ac8c56ed329415fb3101543787153e0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:11:35.968799 28893 consensus_queue.cc:260] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc696ce923844cc1887f8eaa779020ad" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 34247 } }
I20240503 02:11:35.969012 28893 raft_consensus.cc:388] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:35.969009 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80496c7c7d4f4cd69614016a33272eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.969151 28893 raft_consensus.cc:482] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:35.969259 28893 raft_consensus.cc:3037] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.971799 28893 raft_consensus.cc:504] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc696ce923844cc1887f8eaa779020ad" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 34247 } }
I20240503 02:11:35.972007 28893 leader_election.cc:302] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc696ce923844cc1887f8eaa779020ad; no voters:
I20240503 02:11:35.972497 28893 leader_election.cc:288] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:35.972713 28895 raft_consensus.cc:2781] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:35.979728 28893 ts_tablet_manager.cc:1398] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20240503 02:11:35.980429 28896 tablet_bootstrap.cc:492] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: Bootstrap starting.
I20240503 02:11:35.981261 28895 raft_consensus.cc:686] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 1 LEADER]: Becoming Leader. State: Replica: bc696ce923844cc1887f8eaa779020ad, State: Running, Role: LEADER
I20240503 02:11:35.980536 28899 tablet_bootstrap.cc:492] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: Bootstrap starting.
I20240503 02:11:35.981616 28895 consensus_queue.cc:237] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc696ce923844cc1887f8eaa779020ad" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 34247 } }
I20240503 02:11:35.983680 28899 tablet_bootstrap.cc:654] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.983680 28896 tablet_bootstrap.cc:654] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.984505 28409 catalog_manager.cc:5484] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad reported cstate change: term changed from 0 to 1, leader changed from <none> to bc696ce923844cc1887f8eaa779020ad (127.26.178.66). New cstate: current_term: 1 leader_uuid: "bc696ce923844cc1887f8eaa779020ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc696ce923844cc1887f8eaa779020ad" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 34247 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:35.996690 28899 tablet_bootstrap.cc:492] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: No bootstrap required, opened a new log
I20240503 02:11:35.996805 28896 tablet_bootstrap.cc:492] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: No bootstrap required, opened a new log
I20240503 02:11:35.996881 28899 ts_tablet_manager.cc:1367] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240503 02:11:35.997052 28896 ts_tablet_manager.cc:1367] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240503 02:11:35.997694 28899 raft_consensus.cc:348] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 37155 } }
I20240503 02:11:35.997876 28899 raft_consensus.cc:374] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.997871 28896 raft_consensus.cc:348] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44243 } }
I20240503 02:11:35.998008 28899 raft_consensus.cc:729] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63d7e34bd1a4488a2dd7e9ac3216697, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.998061 28896 raft_consensus.cc:374] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.998165 28896 raft_consensus.cc:729] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9793eaecbd4479b47b6b7c4fb22d8f, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.998260 28899 consensus_queue.cc:260] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 37155 } }
I20240503 02:11:35.998476 28899 raft_consensus.cc:388] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:35.998404 28896 consensus_queue.cc:260] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44243 } }
I20240503 02:11:35.998576 28899 raft_consensus.cc:482] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:35.998620 28896 raft_consensus.cc:388] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:35.998733 28896 raft_consensus.cc:482] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:35.998739 28899 raft_consensus.cc:3037] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.998853 28896 raft_consensus.cc:3037] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.000998 28899 raft_consensus.cc:504] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 37155 } }
I20240503 02:11:36.000999 28896 raft_consensus.cc:504] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44243 } }
I20240503 02:11:36.001264 28899 leader_election.cc:302] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f63d7e34bd1a4488a2dd7e9ac3216697; no voters:
I20240503 02:11:36.001287 28896 leader_election.cc:302] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9793eaecbd4479b47b6b7c4fb22d8f; no voters:
I20240503 02:11:36.001907 28899 leader_election.cc:288] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:36.002130 28904 raft_consensus.cc:2781] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:36.002887 28896 leader_election.cc:288] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:36.002938 28899 ts_tablet_manager.cc:1398] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:36.003232 28905 raft_consensus.cc:2781] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:36.003755 28904 raft_consensus.cc:686] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 1 LEADER]: Becoming Leader. State: Replica: f63d7e34bd1a4488a2dd7e9ac3216697, State: Running, Role: LEADER
I20240503 02:11:36.004038 28896 ts_tablet_manager.cc:1398] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20240503 02:11:36.004340 28904 consensus_queue.cc:237] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 37155 } }
I20240503 02:11:36.007644 28905 raft_consensus.cc:686] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 1 LEADER]: Becoming Leader. State: Replica: ca9793eaecbd4479b47b6b7c4fb22d8f, State: Running, Role: LEADER
I20240503 02:11:36.007906 28409 catalog_manager.cc:5484] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 reported cstate change: term changed from 0 to 1, leader changed from <none> to f63d7e34bd1a4488a2dd7e9ac3216697 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63d7e34bd1a4488a2dd7e9ac3216697" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 37155 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:36.008034 28905 consensus_queue.cc:237] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44243 } }
I20240503 02:11:36.018195 28409 catalog_manager.cc:5484] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f reported cstate change: term changed from 0 to 1, leader changed from <none> to ca9793eaecbd4479b47b6b7c4fb22d8f (127.26.178.67). New cstate: current_term: 1 leader_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9793eaecbd4479b47b6b7c4fb22d8f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44243 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:36.037040 27337 master.cc:537] Master@127.26.178.126:37647 shutting down...
I20240503 02:11:36.043998 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:36.044466 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:36.044605 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6c0f3c7fc0564b9f9af72ded16fd60ae: stopping tablet replica
I20240503 02:11:36.049916 27337 master.cc:552] Master@127.26.178.126:37647 shutdown complete.
W20240503 02:11:36.999240 28780 heartbeater.cc:644] Failed to heartbeat to 127.26.178.126:37647 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:37.524528 28918 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c0f3c7fc0564b9f9af72ded16fd60ae)
I20240503 02:11:37.524542 28917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c0f3c7fc0564b9f9af72ded16fd60ae)
I20240503 02:11:37.524772 28918 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:37.524783 28917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:37.525498 28917 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695)
I20240503 02:11:37.525697 28918 leader_election.cc:288] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), 2d4c13ce504b4085b63ee5cff200519a (127.26.178.124:43637)
I20240503 02:11:37.526347 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d4c13ce504b4085b63ee5cff200519a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c" is_pre_election: true
I20240503 02:11:37.526417 28549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9dfd4f415274063b8c239ba4468e31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2d4c13ce504b4085b63ee5cff200519a" is_pre_election: true
I20240503 02:11:37.526646 28549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9dfd4f415274063b8c239ba4468e31c in term 1.
I20240503 02:11:37.526643 28483 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d4c13ce504b4085b63ee5cff200519a in term 1.
I20240503 02:11:37.527406 28460 leader_election.cc:302] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a, c9dfd4f415274063b8c239ba4468e31c; no voters:
I20240503 02:11:37.527544 28522 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a, c9dfd4f415274063b8c239ba4468e31c; no voters:
I20240503 02:11:37.527776 28918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:37.527913 28918 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c0f3c7fc0564b9f9af72ded16fd60ae)
I20240503 02:11:37.528137 28918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:37.527915 28917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 02:11:37.528523 28459 leader_election.cc:334] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:37.528697 28917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c0f3c7fc0564b9f9af72ded16fd60ae)
I20240503 02:11:37.528824 28917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:37.529247 28523 leader_election.cc:334] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:37.578814 28918 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:37.578876 28917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:37.579919 28918 leader_election.cc:288] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 election: Requested vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), 2d4c13ce504b4085b63ee5cff200519a (127.26.178.124:43637)
I20240503 02:11:37.580277 28549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9dfd4f415274063b8c239ba4468e31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2d4c13ce504b4085b63ee5cff200519a"
I20240503 02:11:37.580348 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d4c13ce504b4085b63ee5cff200519a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c"
I20240503 02:11:37.580613 28549 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c9dfd4f415274063b8c239ba4468e31c in current term 2: Already voted for candidate 2d4c13ce504b4085b63ee5cff200519a in this term.
I20240503 02:11:37.580610 28483 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d4c13ce504b4085b63ee5cff200519a in current term 2: Already voted for candidate c9dfd4f415274063b8c239ba4468e31c in this term.
I20240503 02:11:37.580279 28917 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 election: Requested vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695)
W20240503 02:11:37.582376 28523 leader_election.cc:334] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
W20240503 02:11:37.582376 28459 leader_election.cc:334] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:37.582564 28523 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a; no voters: 6c0f3c7fc0564b9f9af72ded16fd60ae, c9dfd4f415274063b8c239ba4468e31c
I20240503 02:11:37.582623 28459 leader_election.cc:302] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9dfd4f415274063b8c239ba4468e31c; no voters: 2d4c13ce504b4085b63ee5cff200519a, 6c0f3c7fc0564b9f9af72ded16fd60ae
I20240503 02:11:37.582924 28917 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:11:37.583148 28918 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:11:39.522986 28929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.523236 28929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:39.523944 28929 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695)
I20240503 02:11:39.524523 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d4c13ce504b4085b63ee5cff200519a" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c" is_pre_election: true
I20240503 02:11:39.524755 28483 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d4c13ce504b4085b63ee5cff200519a in term 2.
W20240503 02:11:39.524938 28523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:11:39.525221 28522 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a, c9dfd4f415274063b8c239ba4468e31c; no voters:
I20240503 02:11:39.525476 28929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:11:39.525612 28929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.525720 28929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:11:39.526428 28523 leader_election.cc:334] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:39.569772 28929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:39.570597 28929 leader_election.cc:288] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 election: Requested vote from peers 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647), c9dfd4f415274063b8c239ba4468e31c (127.26.178.125:41695)
I20240503 02:11:39.571333 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d4c13ce504b4085b63ee5cff200519a" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c9dfd4f415274063b8c239ba4468e31c"
I20240503 02:11:39.571547 28483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:11:39.572319 28523 leader_election.cc:334] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111)
I20240503 02:11:39.669555 28483 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d4c13ce504b4085b63ee5cff200519a in term 3.
I20240503 02:11:39.670305 28522 leader_election.cc:302] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d4c13ce504b4085b63ee5cff200519a, c9dfd4f415274063b8c239ba4468e31c; no voters: 6c0f3c7fc0564b9f9af72ded16fd60ae
I20240503 02:11:39.670637 28929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:11:39.671061 28929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 3 LEADER]: Becoming Leader. State: Replica: 2d4c13ce504b4085b63ee5cff200519a, State: Running, Role: LEADER
I20240503 02:11:39.671350 28929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } }
I20240503 02:11:39.675848 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d4c13ce504b4085b63ee5cff200519a. Latest consensus state: current_term: 3 leader_uuid: "2d4c13ce504b4085b63ee5cff200519a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:39.676142 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:39.676714 28934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:39.678659 28934 catalog_manager.cc:669] Loaded metadata for table test-workload [id=5ac8c56ed329415fb3101543787153e0]
I20240503 02:11:39.680218 28934 tablet_loader.cc:96] loaded metadata for tablet 67cee140a3e8458f8294211ac093d121 (table test-workload [id=5ac8c56ed329415fb3101543787153e0])
I20240503 02:11:39.680601 28934 tablet_loader.cc:96] loaded metadata for tablet 80496c7c7d4f4cd69614016a33272eb4 (table test-workload [id=5ac8c56ed329415fb3101543787153e0])
I20240503 02:11:39.680953 28934 tablet_loader.cc:96] loaded metadata for tablet 9eeaf3e2329d4e91a5fe215600d0245f (table test-workload [id=5ac8c56ed329415fb3101543787153e0])
I20240503 02:11:39.681200 28934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:39.682083 28934 catalog_manager.cc:1259] Loaded cluster ID: cb86c775dde94af0b361114c71bd3de8
I20240503 02:11:39.682183 28934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:39.683466 28934 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:39.684383 28934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: Loaded TSK: 0
I20240503 02:11:39.685094 28934 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 02:11:40.066593 28523 consensus_peers.cc:485] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a -> Peer 6c0f3c7fc0564b9f9af72ded16fd60ae (127.26.178.126:37647): Couldn't send request to peer 6c0f3c7fc0564b9f9af72ded16fd60ae. Status: Network error: Client connection negotiation failed: client connection to 127.26.178.126:37647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:40.291497 28483 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 3 FOLLOWER]: Refusing update from remote peer 2d4c13ce504b4085b63ee5cff200519a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:11:40.292177 28932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:11:40.294015 28938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d4c13ce504b4085b63ee5cff200519a. Latest consensus state: current_term: 3 leader_uuid: "2d4c13ce504b4085b63ee5cff200519a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:40.294241 28938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:40.295832 28940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2d4c13ce504b4085b63ee5cff200519a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:40.296044 28940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:40.296478 28938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2d4c13ce504b4085b63ee5cff200519a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0f3c7fc0564b9f9af72ded16fd60ae" member_type: VOTER last_known_addr { host: "127.26.178.126" port: 37647 } } peers { permanent_uuid: "c9dfd4f415274063b8c239ba4468e31c" member_type: VOTER last_known_addr { host: "127.26.178.125" port: 41695 } } peers { permanent_uuid: "2d4c13ce504b4085b63ee5cff200519a" member_type: VOTER last_known_addr { host: "127.26.178.124" port: 43637 } } }
I20240503 02:11:40.296916 28938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:11:40.997011 28781 heartbeater.cc:503] Master 127.26.178.124:43637 was elected leader, sending a full tablet report...
I20240503 02:11:41.021942 28690 heartbeater.cc:503] Master 127.26.178.124:43637 was elected leader, sending a full tablet report...
I20240503 02:11:41.026479 28859 heartbeater.cc:503] Master 127.26.178.124:43637 was elected leader, sending a full tablet report...
I20240503 02:11:41.606266 28607 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:41.606444 28607 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:41.800966 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:11:41.811242 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:41.811520 27337 tablet_replica.cc:330] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697: stopping tablet replica
I20240503 02:11:41.811733 27337 raft_consensus.cc:2229] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:41.811910 27337 raft_consensus.cc:2258] T 80496c7c7d4f4cd69614016a33272eb4 P f63d7e34bd1a4488a2dd7e9ac3216697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:41.825738 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:11:41.828866 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:11:41.838074 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:41.838491 27337 tablet_replica.cc:330] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad: stopping tablet replica
I20240503 02:11:41.838794 27337 raft_consensus.cc:2229] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:41.839052 27337 raft_consensus.cc:2258] T 67cee140a3e8458f8294211ac093d121 P bc696ce923844cc1887f8eaa779020ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:41.852726 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:11:41.855484 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:11:41.863782 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:11:41.864050 27337 tablet_replica.cc:330] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f: stopping tablet replica
I20240503 02:11:41.864240 27337 raft_consensus.cc:2229] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:11:41.864410 27337 raft_consensus.cc:2258] T 9eeaf3e2329d4e91a5fe215600d0245f P ca9793eaecbd4479b47b6b7c4fb22d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:41.878103 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:11:41.881121 27337 master.cc:537] Master@127.26.178.125:41695 shutting down...
I20240503 02:11:41.889125 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:41.889410 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:41.889569 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P c9dfd4f415274063b8c239ba4468e31c: stopping tablet replica
I20240503 02:11:41.903514 27337 master.cc:552] Master@127.26.178.125:41695 shutdown complete.
I20240503 02:11:41.908089 27337 master.cc:537] Master@127.26.178.124:43637 shutting down...
I20240503 02:11:41.915143 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:11:41.915511 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:41.915609 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2d4c13ce504b4085b63ee5cff200519a: stopping tablet replica
I20240503 02:11:41.929075 27337 master.cc:552] Master@127.26.178.124:43637 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (8507 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 02:11:41.942183 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:44221
I20240503 02:11:41.942767 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.946077 28949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.947177 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.947980 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.948786 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:41.949079 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:41.949141 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:41.949208 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702301949163 us; error 0 us; skew 500 ppm
I20240503 02:11:41.949373 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.950901 27337 webserver.cc:460] Webserver started at http://127.26.178.126:33911/ using document root <none> and password file <none>
I20240503 02:11:41.951148 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.951263 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.951423 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:41.952209 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "02083d5e81674180a6800545e362c1bf"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-0cbc"
I20240503 02:11:41.954849 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 02:11:41.956719 28957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.957113 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:41.957242 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "02083d5e81674180a6800545e362c1bf"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-0cbc"
I20240503 02:11:41.957418 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.986348 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.987082 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.001201 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:44221
I20240503 02:11:42.001256 29009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:44221 every 8 connection(s)
I20240503 02:11:42.002562 29010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:42.006582 29010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf: Bootstrap starting.
I20240503 02:11:42.007894 29010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:42.009677 29010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf: No bootstrap required, opened a new log
I20240503 02:11:42.010438 29010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER }
I20240503 02:11:42.010578 29010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:42.010648 29010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02083d5e81674180a6800545e362c1bf, State: Initialized, Role: FOLLOWER
I20240503 02:11:42.010901 29010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER }
I20240503 02:11:42.011058 29010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:42.011137 29010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:42.011251 29010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:42.012948 29010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER }
I20240503 02:11:42.013150 29010 leader_election.cc:302] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02083d5e81674180a6800545e362c1bf; no voters:
I20240503 02:11:42.013576 29010 leader_election.cc:288] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:42.013728 29013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:42.014199 29013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 1 LEADER]: Becoming Leader. State: Replica: 02083d5e81674180a6800545e362c1bf, State: Running, Role: LEADER
I20240503 02:11:42.014490 29010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:42.014510 29013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER }
I20240503 02:11:42.015461 29014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02083d5e81674180a6800545e362c1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER } }
I20240503 02:11:42.015558 29015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02083d5e81674180a6800545e362c1bf. Latest consensus state: current_term: 1 leader_uuid: "02083d5e81674180a6800545e362c1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02083d5e81674180a6800545e362c1bf" member_type: VOTER } }
I20240503 02:11:42.015743 29014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:42.015844 29015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:42.016834 29018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:42.018143 29018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:42.021572 29018 catalog_manager.cc:1347] Generated new cluster ID: 34f364b62d294977a74e9c20b49b56e3
I20240503 02:11:42.021673 29018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:42.021950 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:11:42.031813 29018 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:42.032647 29018 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:42.044490 29018 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf: Generated new TSK 0
I20240503 02:11:42.044759 29018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:42.054899 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.058764 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.059744 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.060587 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.061506 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:42.061780 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:42.061866 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:42.061928 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702302061907 us; error 0 us; skew 500 ppm
I20240503 02:11:42.062091 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.063637 27337 webserver.cc:460] Webserver started at http://127.26.178.65:33099/ using document root <none> and password file <none>
I20240503 02:11:42.063897 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.063995 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.064191 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.064867 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "0fc954cb8e92453bb31279a0a7e40003"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.067416 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:11:42.069141 29039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.069561 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:42.069684 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "0fc954cb8e92453bb31279a0a7e40003"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.069803 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.090596 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.091317 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.091933 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.092875 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.092967 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.093072 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.093148 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.108554 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:36195
I20240503 02:11:42.108599 29101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:36195 every 8 connection(s)
I20240503 02:11:42.110422 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.114871 29106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.115896 29107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.116513 29109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.117630 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:42.117952 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:42.118031 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:42.118110 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702302118073 us; error 0 us; skew 500 ppm
I20240503 02:11:42.118311 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.118337 29102 heartbeater.cc:347] Connected to a master server at 127.26.178.126:44221
I20240503 02:11:42.118476 29102 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.119103 29102 heartbeater.cc:511] Master 127.26.178.126:44221 requested a full tablet report, sending...
I20240503 02:11:42.119944 27337 webserver.cc:460] Webserver started at http://127.26.178.66:33337/ using document root <none> and password file <none>
I20240503 02:11:42.120203 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.120313 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.120352 28974 ts_manager.cc:194] Registered new tserver with Master: 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65:36195)
I20240503 02:11:42.120472 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.121166 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "145b864767da4dd49712dcb54292c5fc"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.121536 28974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42548
I20240503 02:11:42.124020 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:42.133805 29115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.134240 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:42.134349 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "145b864767da4dd49712dcb54292c5fc"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.134475 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.144830 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.145323 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.145803 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.146590 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.146648 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.146764 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.146859 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.158164 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:35145
I20240503 02:11:42.158200 29177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:35145 every 8 connection(s)
I20240503 02:11:42.159641 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.163298 29181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.164003 29182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.165156 29184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.166857 29178 heartbeater.cc:347] Connected to a master server at 127.26.178.126:44221
I20240503 02:11:42.166987 29178 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.167372 29178 heartbeater.cc:511] Master 127.26.178.126:44221 requested a full tablet report, sending...
I20240503 02:11:42.168207 28974 ts_manager.cc:194] Registered new tserver with Master: 145b864767da4dd49712dcb54292c5fc (127.26.178.66:35145)
I20240503 02:11:42.168803 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:42.169077 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:11:42.169095 28974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42562
W20240503 02:11:42.169129 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:42.169170 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702302169165 us; error 0 us; skew 500 ppm
I20240503 02:11:42.169292 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.170662 27337 webserver.cc:460] Webserver started at http://127.26.178.67:45559/ using document root <none> and password file <none>
I20240503 02:11:42.170899 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.170964 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.171068 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.171645 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "ac2926385c7d42ffa36ddd57ab08ebc5"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.173827 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:42.175355 29189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.175654 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:11:42.175745 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "ac2926385c7d42ffa36ddd57ab08ebc5"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-0cbc"
I20240503 02:11:42.175840 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.185096 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.185667 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.186193 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.187007 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.187063 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.187134 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.187168 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.201527 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:39015
I20240503 02:11:42.201557 29252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:39015 every 8 connection(s)
I20240503 02:11:42.208060 29253 heartbeater.cc:347] Connected to a master server at 127.26.178.126:44221
I20240503 02:11:42.208207 29253 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.208505 29253 heartbeater.cc:511] Master 127.26.178.126:44221 requested a full tablet report, sending...
I20240503 02:11:42.209328 28974 ts_manager.cc:194] Registered new tserver with Master: ac2926385c7d42ffa36ddd57ab08ebc5 (127.26.178.67:39015)
I20240503 02:11:42.209995 28974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42568
I20240503 02:11:42.210290 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007910456s
I20240503 02:11:43.123034 29102 heartbeater.cc:503] Master 127.26.178.126:44221 was elected leader, sending a full tablet report...
I20240503 02:11:43.170269 29178 heartbeater.cc:503] Master 127.26.178.126:44221 was elected leader, sending a full tablet report...
I20240503 02:11:43.211180 29253 heartbeater.cc:503] Master 127.26.178.126:44221 was elected leader, sending a full tablet report...
I20240503 02:11:43.237957 27337 test_util.cc:268] Using random seed: 1034819389
I20240503 02:11:43.249836 28974 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42584:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:11:43.250514 28974 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:11:43.280443 29217 tablet_service.cc:1467] Processing CreateTablet for tablet beb281c36aa04c239347373cf1ad9109 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:11:43.281101 29217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb281c36aa04c239347373cf1ad9109. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.281157 29142 tablet_service.cc:1467] Processing CreateTablet for tablet 33618faa82f242b984cb6aa123115462 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:11:43.281128 29143 tablet_service.cc:1467] Processing CreateTablet for tablet beb281c36aa04c239347373cf1ad9109 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:11:43.281689 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb281c36aa04c239347373cf1ad9109. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.281872 29142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33618faa82f242b984cb6aa123115462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.286338 29273 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: Bootstrap starting.
I20240503 02:11:43.287358 29216 tablet_service.cc:1467] Processing CreateTablet for tablet 33618faa82f242b984cb6aa123115462 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:11:43.287865 29216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33618faa82f242b984cb6aa123115462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.288386 29273 tablet_bootstrap.cc:654] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.288396 29067 tablet_service.cc:1467] Processing CreateTablet for tablet beb281c36aa04c239347373cf1ad9109 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:11:43.288950 29067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb281c36aa04c239347373cf1ad9109. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.288908 29066 tablet_service.cc:1467] Processing CreateTablet for tablet 33618faa82f242b984cb6aa123115462 (DEFAULT_TABLE table=test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:11:43.289307 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33618faa82f242b984cb6aa123115462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.300413 29274 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: Bootstrap starting.
I20240503 02:11:43.303455 29276 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: Bootstrap starting.
I20240503 02:11:43.304855 29276 tablet_bootstrap.cc:654] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.305837 29274 tablet_bootstrap.cc:654] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.307909 29276 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: No bootstrap required, opened a new log
I20240503 02:11:43.307992 29273 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: No bootstrap required, opened a new log
I20240503 02:11:43.308041 29276 ts_tablet_manager.cc:1367] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:43.308145 29273 ts_tablet_manager.cc:1367] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.011s
I20240503 02:11:43.308627 29276 raft_consensus.cc:348] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.308777 29276 raft_consensus.cc:374] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.308811 29276 raft_consensus.cc:729] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc954cb8e92453bb31279a0a7e40003, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.308699 29273 raft_consensus.cc:348] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.308894 29273 raft_consensus.cc:374] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.309000 29273 raft_consensus.cc:729] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2926385c7d42ffa36ddd57ab08ebc5, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.309010 29276 consensus_queue.cc:260] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.309253 29273 consensus_queue.cc:260] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.309597 29274 tablet_bootstrap.cc:492] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: No bootstrap required, opened a new log
I20240503 02:11:43.309782 29274 ts_tablet_manager.cc:1367] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 02:11:43.310197 29273 ts_tablet_manager.cc:1398] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:43.310468 29274 raft_consensus.cc:348] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.310597 29273 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: Bootstrap starting.
I20240503 02:11:43.310748 29274 raft_consensus.cc:374] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.310873 29274 raft_consensus.cc:729] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145b864767da4dd49712dcb54292c5fc, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.311157 29274 consensus_queue.cc:260] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.312656 29273 tablet_bootstrap.cc:654] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.316062 29274 ts_tablet_manager.cc:1398] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240503 02:11:43.316711 29276 ts_tablet_manager.cc:1398] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20240503 02:11:43.316787 29274 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: Bootstrap starting.
I20240503 02:11:43.317723 29276 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Bootstrap starting.
I20240503 02:11:43.318812 29274 tablet_bootstrap.cc:654] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.319890 29276 tablet_bootstrap.cc:654] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.321981 29273 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: No bootstrap required, opened a new log
I20240503 02:11:43.322137 29273 ts_tablet_manager.cc:1367] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20240503 02:11:43.322446 29276 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: No bootstrap required, opened a new log
I20240503 02:11:43.322507 29274 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: No bootstrap required, opened a new log
I20240503 02:11:43.322568 29276 ts_tablet_manager.cc:1367] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:11:43.322722 29274 ts_tablet_manager.cc:1367] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20240503 02:11:43.322938 29273 raft_consensus.cc:348] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.323149 29273 raft_consensus.cc:374] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.323119 29276 raft_consensus.cc:348] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.323257 29273 raft_consensus.cc:729] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2926385c7d42ffa36ddd57ab08ebc5, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.323304 29276 raft_consensus.cc:374] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.323393 29276 raft_consensus.cc:729] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc954cb8e92453bb31279a0a7e40003, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.323416 29274 raft_consensus.cc:348] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.323591 29274 raft_consensus.cc:374] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.323652 29274 raft_consensus.cc:729] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145b864767da4dd49712dcb54292c5fc, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.323561 29273 consensus_queue.cc:260] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.323606 29276 consensus_queue.cc:260] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.323906 29274 consensus_queue.cc:260] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.324122 29276 ts_tablet_manager.cc:1398] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:11:43.324440 29274 ts_tablet_manager.cc:1398] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:43.324152 29273 ts_tablet_manager.cc:1398] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:43.362913 29280 raft_consensus.cc:482] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.363092 29280 raft_consensus.cc:504] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.363804 29280 leader_election.cc:288] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65:36195), 145b864767da4dd49712dcb54292c5fc (127.26.178.66:35145)
I20240503 02:11:43.371376 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33618faa82f242b984cb6aa123115462" candidate_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145b864767da4dd49712dcb54292c5fc" is_pre_election: true
I20240503 02:11:43.371399 29077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33618faa82f242b984cb6aa123115462" candidate_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc954cb8e92453bb31279a0a7e40003" is_pre_election: true
I20240503 02:11:43.371645 29153 raft_consensus.cc:2454] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac2926385c7d42ffa36ddd57ab08ebc5 in term 0.
I20240503 02:11:43.371655 29077 raft_consensus.cc:2454] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac2926385c7d42ffa36ddd57ab08ebc5 in term 0.
I20240503 02:11:43.372066 29193 leader_election.cc:302] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145b864767da4dd49712dcb54292c5fc, ac2926385c7d42ffa36ddd57ab08ebc5; no voters:
I20240503 02:11:43.372331 29280 raft_consensus.cc:2781] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.372418 29280 raft_consensus.cc:482] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.372496 29280 raft_consensus.cc:3037] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.374300 29280 raft_consensus.cc:504] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.374883 29280 leader_election.cc:288] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [CANDIDATE]: Term 1 election: Requested vote from peers 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65:36195), 145b864767da4dd49712dcb54292c5fc (127.26.178.66:35145)
I20240503 02:11:43.375358 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33618faa82f242b984cb6aa123115462" candidate_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145b864767da4dd49712dcb54292c5fc"
I20240503 02:11:43.375358 29077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33618faa82f242b984cb6aa123115462" candidate_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc954cb8e92453bb31279a0a7e40003"
I20240503 02:11:43.375561 29077 raft_consensus.cc:3037] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.375561 29153 raft_consensus.cc:3037] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.378399 29077 raft_consensus.cc:2454] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac2926385c7d42ffa36ddd57ab08ebc5 in term 1.
I20240503 02:11:43.378399 29153 raft_consensus.cc:2454] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac2926385c7d42ffa36ddd57ab08ebc5 in term 1.
I20240503 02:11:43.378794 29193 leader_election.cc:302] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145b864767da4dd49712dcb54292c5fc, ac2926385c7d42ffa36ddd57ab08ebc5; no voters:
I20240503 02:11:43.379042 29280 raft_consensus.cc:2781] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.379547 29280 raft_consensus.cc:686] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 LEADER]: Becoming Leader. State: Replica: ac2926385c7d42ffa36ddd57ab08ebc5, State: Running, Role: LEADER
I20240503 02:11:43.379802 29280 consensus_queue.cc:237] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } }
I20240503 02:11:43.382305 28972 catalog_manager.cc:5484] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac2926385c7d42ffa36ddd57ab08ebc5 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:43.436044 29279 raft_consensus.cc:482] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.436221 29279 raft_consensus.cc:504] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.436955 29279 leader_election.cc:288] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac2926385c7d42ffa36ddd57ab08ebc5 (127.26.178.67:39015), 145b864767da4dd49712dcb54292c5fc (127.26.178.66:35145)
I20240503 02:11:43.442868 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb281c36aa04c239347373cf1ad9109" candidate_uuid: "0fc954cb8e92453bb31279a0a7e40003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145b864767da4dd49712dcb54292c5fc" is_pre_election: true
I20240503 02:11:43.442874 29227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb281c36aa04c239347373cf1ad9109" candidate_uuid: "0fc954cb8e92453bb31279a0a7e40003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" is_pre_election: true
I20240503 02:11:43.443122 29153 raft_consensus.cc:2454] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc954cb8e92453bb31279a0a7e40003 in term 0.
I20240503 02:11:43.443176 29227 raft_consensus.cc:2454] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc954cb8e92453bb31279a0a7e40003 in term 0.
I20240503 02:11:43.443480 29043 leader_election.cc:302] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc954cb8e92453bb31279a0a7e40003, 145b864767da4dd49712dcb54292c5fc; no voters:
I20240503 02:11:43.443722 29279 raft_consensus.cc:2781] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.443789 29279 raft_consensus.cc:482] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.443846 29279 raft_consensus.cc:3037] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.445724 29279 raft_consensus.cc:504] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.446317 29279 leader_election.cc:288] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [CANDIDATE]: Term 1 election: Requested vote from peers ac2926385c7d42ffa36ddd57ab08ebc5 (127.26.178.67:39015), 145b864767da4dd49712dcb54292c5fc (127.26.178.66:35145)
I20240503 02:11:43.446722 29227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb281c36aa04c239347373cf1ad9109" candidate_uuid: "0fc954cb8e92453bb31279a0a7e40003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5"
I20240503 02:11:43.446764 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb281c36aa04c239347373cf1ad9109" candidate_uuid: "0fc954cb8e92453bb31279a0a7e40003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145b864767da4dd49712dcb54292c5fc"
I20240503 02:11:43.446911 29227 raft_consensus.cc:3037] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.446939 29153 raft_consensus.cc:3037] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.448895 29153 raft_consensus.cc:2454] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc954cb8e92453bb31279a0a7e40003 in term 1.
I20240503 02:11:43.449018 29227 raft_consensus.cc:2454] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc954cb8e92453bb31279a0a7e40003 in term 1.
I20240503 02:11:43.449270 29043 leader_election.cc:302] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc954cb8e92453bb31279a0a7e40003, 145b864767da4dd49712dcb54292c5fc; no voters:
I20240503 02:11:43.449545 29279 raft_consensus.cc:2781] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.449995 29279 raft_consensus.cc:686] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc954cb8e92453bb31279a0a7e40003, State: Running, Role: LEADER
I20240503 02:11:43.450273 29279 consensus_queue.cc:237] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } }
I20240503 02:11:43.452797 28972 catalog_manager.cc:5484] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "0fc954cb8e92453bb31279a0a7e40003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:43.771458 29280 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:43.777640 29296 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:43.945469 29279 consensus_queue.cc:1035] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:43.949754 29279 consensus_queue.cc:1035] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.021876 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:44.022478 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:44.022635 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:46.023113 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:46.023880 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:46.024022 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:48.024557 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:48.025215 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:48.025382 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:50.025849 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:50.026521 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:50.026656 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:52.027215 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:52.027907 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:52.028052 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:54.028556 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:54.029145 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:54.029263 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:54.526293 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:54.530436 29303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.531307 29304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.532270 29306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:54.533121 27337 server_base.cc:1034] running on GCE node
I20240503 02:11:54.533425 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:11:54.533495 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:11:54.533555 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702314533521 us; error 0 us; skew 500 ppm
I20240503 02:11:54.533725 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:54.535432 27337 webserver.cc:460] Webserver started at http://127.26.178.68:42379/ using document root <none> and password file <none>
I20240503 02:11:54.535679 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:54.535780 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:54.535928 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:54.536639 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-3-root/instance:
uuid: "078d19a67eb34c9eb9fdc994d7e03825"
format_stamp: "Formatted at 2024-05-03 02:11:54 on dist-test-slave-0cbc"
I20240503 02:11:54.539640 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:11:54.541374 29311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.541872 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:54.541991 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-3-root
uuid: "078d19a67eb34c9eb9fdc994d7e03825"
format_stamp: "Formatted at 2024-05-03 02:11:54 on dist-test-slave-0cbc"
I20240503 02:11:54.542189 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:54.570531 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:54.571259 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:54.571842 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:54.572770 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:54.572835 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.572925 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:54.572988 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.589107 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.68:39257
I20240503 02:11:54.589138 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.68:39257 every 8 connection(s)
I20240503 02:11:54.596498 29374 heartbeater.cc:347] Connected to a master server at 127.26.178.126:44221
I20240503 02:11:54.596693 29374 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:54.597131 29374 heartbeater.cc:511] Master 127.26.178.126:44221 requested a full tablet report, sending...
I20240503 02:11:54.598186 28972 ts_manager.cc:194] Registered new tserver with Master: 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68:39257)
I20240503 02:11:54.599128 28972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59998
I20240503 02:11:55.044875 29227 consensus_queue.cc:237] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } }
I20240503 02:11:55.047474 29077 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:55.048085 29153 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:55.048266 29302 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:55.048774 29383 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:11:55.051271 29302 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } } }
I20240503 02:11:55.052111 29153 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } } }
I20240503 02:11:55.052978 29077 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } } }
I20240503 02:11:55.057832 28972 catalog_manager.cc:5484] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:55.059295 29193 consensus_peers.cc:485] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 -> Peer 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68:39257): Couldn't send request to peer 078d19a67eb34c9eb9fdc994d7e03825. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33618faa82f242b984cb6aa123115462. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:55.525110 29393 ts_tablet_manager.cc:902] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Initiating tablet copy from peer ac2926385c7d42ffa36ddd57ab08ebc5 (127.26.178.67:39015)
I20240503 02:11:55.526983 29393 tablet_copy_client.cc:308] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:39015
I20240503 02:11:55.533681 29237 tablet_copy_service.cc:140] P ac2926385c7d42ffa36ddd57ab08ebc5: Received BeginTabletCopySession request for tablet 33618faa82f242b984cb6aa123115462 from peer 078d19a67eb34c9eb9fdc994d7e03825 ({username='slave'} at 127.0.0.1:54206)
I20240503 02:11:55.533911 29237 tablet_copy_service.cc:161] P ac2926385c7d42ffa36ddd57ab08ebc5: Beginning new tablet copy session on tablet 33618faa82f242b984cb6aa123115462 from peer 078d19a67eb34c9eb9fdc994d7e03825 at {username='slave'} at 127.0.0.1:54206: session id = 078d19a67eb34c9eb9fdc994d7e03825-33618faa82f242b984cb6aa123115462
I20240503 02:11:55.537305 29237 tablet_copy_source_session.cc:205] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:55.539299 29393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33618faa82f242b984cb6aa123115462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:55.545014 29393 tablet_copy_client.cc:783] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:55.545321 29393 tablet_copy_client.cc:647] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:55.547463 29393 tablet_copy_client.cc:523] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:55.551178 29393 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Bootstrap starting.
I20240503 02:11:55.558034 29393 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:55.558465 29393 tablet_bootstrap.cc:492] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Bootstrap complete.
I20240503 02:11:55.558737 29393 ts_tablet_manager.cc:1367] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 02:11:55.559391 29393 raft_consensus.cc:348] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } }
I20240503 02:11:55.559577 29393 raft_consensus.cc:729] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 078d19a67eb34c9eb9fdc994d7e03825, State: Initialized, Role: LEARNER
I20240503 02:11:55.559753 29393 consensus_queue.cc:260] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: true } }
I20240503 02:11:55.560585 29393 ts_tablet_manager.cc:1398] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:55.560876 29374 heartbeater.cc:503] Master 127.26.178.126:44221 was elected leader, sending a full tablet report...
I20240503 02:11:55.561654 29237 tablet_copy_service.cc:342] P ac2926385c7d42ffa36ddd57ab08ebc5: Request end of tablet copy session 078d19a67eb34c9eb9fdc994d7e03825-33618faa82f242b984cb6aa123115462 received from {username='slave'} at 127.0.0.1:54206
I20240503 02:11:55.561820 29237 tablet_copy_service.cc:433] P ac2926385c7d42ffa36ddd57ab08ebc5: ending tablet copy session 078d19a67eb34c9eb9fdc994d7e03825-33618faa82f242b984cb6aa123115462 on tablet 33618faa82f242b984cb6aa123115462 with peer 078d19a67eb34c9eb9fdc994d7e03825
I20240503 02:11:55.939702 29349 raft_consensus.cc:1203] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:11:56.029564 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:56.030134 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:56.030243 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:11:56.512995 29402 raft_consensus.cc:1050] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: attempting to promote NON_VOTER 078d19a67eb34c9eb9fdc994d7e03825 to VOTER
I20240503 02:11:56.514001 29402 consensus_queue.cc:237] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } }
I20240503 02:11:56.516230 29349 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 LEARNER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:11:56.516788 29403 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:11:56.517026 29153 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:11:56.517582 29403 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:11:56.520970 29403 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.521546 29153 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.528558 29349 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.529440 28973 catalog_manager.cc:5484] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc reported cstate change: config changed from index 2 to 3, 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.533253 29077 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:11:56.533995 29410 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:11:56.540695 29077 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 078d19a67eb34c9eb9fdc994d7e03825 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fc954cb8e92453bb31279a0a7e40003" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 36195 } attrs { replace: true } } peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.558743 29226 consensus_queue.cc:237] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } }
I20240503 02:11:56.560784 29349 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:11:56.560827 29153 raft_consensus.cc:1261] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Refusing update from remote peer ac2926385c7d42ffa36ddd57ab08ebc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:11:56.561403 29410 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:56.561671 29405 consensus_queue.cc:1035] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:56.564162 29409 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.567102 28958 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33618faa82f242b984cb6aa123115462 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:11:56.567484 29349 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.567601 29153 raft_consensus.cc:2932] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } } }
I20240503 02:11:56.568408 28974 catalog_manager.cc:5484] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 reported cstate change: config changed from index 3 to 4, VOTER 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65) evicted. New cstate: current_term: 1 leader_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "145b864767da4dd49712dcb54292c5fc" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 35145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac2926385c7d42ffa36ddd57ab08ebc5" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078d19a67eb34c9eb9fdc994d7e03825" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 39257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:56.573818 29067 tablet_service.cc:1514] Processing DeleteTablet for tablet 33618faa82f242b984cb6aa123115462 with delete_type TABLET_DATA_TOMBSTONED (TS 0fc954cb8e92453bb31279a0a7e40003 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56540
I20240503 02:11:56.574834 29417 tablet_replica.cc:330] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: stopping tablet replica
I20240503 02:11:56.575028 29417 raft_consensus.cc:2229] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:56.575268 29417 raft_consensus.cc:2258] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:56.576305 29417 ts_tablet_manager.cc:1875] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:11:56.581337 29417 ts_tablet_manager.cc:1888] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:11:56.581470 29417 log.cc:1196] T 33618faa82f242b984cb6aa123115462 P 0fc954cb8e92453bb31279a0a7e40003: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/33618faa82f242b984cb6aa123115462
I20240503 02:11:56.582171 28959 catalog_manager.cc:4830] TS 0fc954cb8e92453bb31279a0a7e40003 (127.26.178.65:36195): tablet 33618faa82f242b984cb6aa123115462 (table test-workload [id=a7a9d0df6acd40b4ad69140f93a69ebc]) successfully deleted
I20240503 02:11:58.030702 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:11:58.031383 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:11:58.031499 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:00.031945 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:00.032647 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:00.032801 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:02.033249 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:02.033905 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:02.034019 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:04.034519 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:04.035445 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:04.035581 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:06.036041 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:06.036724 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:06.036844 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:08.037307 29029 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:08.038020 29029 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:08.038142 29029 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:08.673079 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:12:08.681506 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:08.681824 27337 tablet_replica.cc:330] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003: stopping tablet replica
I20240503 02:12:08.681980 27337 raft_consensus.cc:2229] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:08.682317 27337 raft_consensus.cc:2258] T beb281c36aa04c239347373cf1ad9109 P 0fc954cb8e92453bb31279a0a7e40003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.696317 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:12:08.699460 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:12:08.707965 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:08.708228 27337 tablet_replica.cc:330] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc: stopping tablet replica
I20240503 02:12:08.708403 27337 raft_consensus.cc:2229] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:08.708652 27337 raft_consensus.cc:2258] T 33618faa82f242b984cb6aa123115462 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.709196 27337 tablet_replica.cc:330] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc: stopping tablet replica
I20240503 02:12:08.709300 27337 raft_consensus.cc:2229] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:08.709408 27337 raft_consensus.cc:2258] T beb281c36aa04c239347373cf1ad9109 P 145b864767da4dd49712dcb54292c5fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.723611 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:12:08.726630 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:12:08.735239 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:08.735483 27337 tablet_replica.cc:330] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5: stopping tablet replica
I20240503 02:12:08.735669 27337 raft_consensus.cc:2229] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:08.735965 27337 raft_consensus.cc:2258] T 33618faa82f242b984cb6aa123115462 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.736553 27337 tablet_replica.cc:330] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5: stopping tablet replica
I20240503 02:12:08.736658 27337 raft_consensus.cc:2229] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:08.736768 27337 raft_consensus.cc:2258] T beb281c36aa04c239347373cf1ad9109 P ac2926385c7d42ffa36ddd57ab08ebc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.750584 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:12:08.753772 27337 tablet_server.cc:178] TabletServer@127.26.178.68:0 shutting down...
I20240503 02:12:08.762944 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:08.763242 27337 tablet_replica.cc:330] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825: stopping tablet replica
I20240503 02:12:08.763423 27337 raft_consensus.cc:2229] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:08.763644 27337 raft_consensus.cc:2258] T 33618faa82f242b984cb6aa123115462 P 078d19a67eb34c9eb9fdc994d7e03825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.777081 27337 tablet_server.cc:194] TabletServer@127.26.178.68:0 shutdown complete.
I20240503 02:12:08.780206 27337 master.cc:537] Master@127.26.178.126:44221 shutting down...
I20240503 02:12:08.787381 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:08.787627 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.787719 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02083d5e81674180a6800545e362c1bf: stopping tablet replica
I20240503 02:12:08.800762 27337 master.cc:552] Master@127.26.178.126:44221 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26875 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 02:12:08.817625 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:39287
I20240503 02:12:08.818130 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:08.821956 29423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:08.823065 29424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:08.824049 29426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:08.824996 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:08.825246 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:08.825301 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:08.825338 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702328825334 us; error 0 us; skew 500 ppm
I20240503 02:12:08.825493 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:08.833551 27337 webserver.cc:460] Webserver started at http://127.26.178.126:42405/ using document root <none> and password file <none>
I20240503 02:12:08.833806 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:08.833873 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:08.833997 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:08.834631 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "bf932399269c4108afb6f31771339f94"
format_stamp: "Formatted at 2024-05-03 02:12:08 on dist-test-slave-0cbc"
I20240503 02:12:08.837102 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:08.838732 29431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:08.839126 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:08.839260 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "bf932399269c4108afb6f31771339f94"
format_stamp: "Formatted at 2024-05-03 02:12:08 on dist-test-slave-0cbc"
I20240503 02:12:08.839366 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:08.854069 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:08.854574 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:08.864977 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:39287
I20240503 02:12:08.865007 29482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:39287 every 8 connection(s)
I20240503 02:12:08.866233 29483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.870282 29483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94: Bootstrap starting.
I20240503 02:12:08.871429 29483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.872887 29483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94: No bootstrap required, opened a new log
I20240503 02:12:08.873462 29483 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER }
I20240503 02:12:08.873584 29483 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.873616 29483 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf932399269c4108afb6f31771339f94, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.873803 29483 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER }
I20240503 02:12:08.873904 29483 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:08.873960 29483 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:08.874032 29483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:08.875857 29483 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER }
I20240503 02:12:08.876027 29483 leader_election.cc:302] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf932399269c4108afb6f31771339f94; no voters:
I20240503 02:12:08.876480 29483 leader_election.cc:288] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:08.876601 29486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:08.877051 29486 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 1 LEADER]: Becoming Leader. State: Replica: bf932399269c4108afb6f31771339f94, State: Running, Role: LEADER
I20240503 02:12:08.877393 29483 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:08.877339 29486 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER }
I20240503 02:12:08.878192 29487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf932399269c4108afb6f31771339f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER } }
I20240503 02:12:08.878248 29488 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf932399269c4108afb6f31771339f94. Latest consensus state: current_term: 1 leader_uuid: "bf932399269c4108afb6f31771339f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf932399269c4108afb6f31771339f94" member_type: VOTER } }
I20240503 02:12:08.878408 29487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:08.878537 29488 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:08.879638 29491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:08.880797 29491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:08.882378 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:12:08.883492 29491 catalog_manager.cc:1347] Generated new cluster ID: 30722002d14746a1ac31bfe0ac971ec8
I20240503 02:12:08.883602 29491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:08.891139 29491 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:08.891919 29491 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:08.898468 29491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94: Generated new TSK 0
I20240503 02:12:08.898782 29491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:08.914744 27337 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 9e-05s
I20240503 02:12:20.997931 27337 master.cc:537] Master@127.26.178.126:39287 shutting down...
I20240503 02:12:21.009732 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:21.010006 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.010138 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P bf932399269c4108afb6f31771339f94: stopping tablet replica
I20240503 02:12:21.023898 27337 master.cc:552] Master@127.26.178.126:39287 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12214 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 02:12:21.032297 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:37987
I20240503 02:12:21.032840 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:21.036553 29506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.037428 29505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.038359 29508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.039203 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:21.039459 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:21.039521 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:21.039577 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702341039559 us; error 0 us; skew 500 ppm
I20240503 02:12:21.039752 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:21.049726 27337 webserver.cc:460] Webserver started at http://127.26.178.126:41499/ using document root <none> and password file <none>
I20240503 02:12:21.050011 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:21.050102 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:21.050249 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:21.051043 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "a5d37cce7fba4462a6c9b1b060f68077"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.053644 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:21.055400 29513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.055769 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:12:21.055874 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "a5d37cce7fba4462a6c9b1b060f68077"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.056016 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:21.084125 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:21.084771 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:21.096822 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:37987
I20240503 02:12:21.096834 29564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:37987 every 8 connection(s)
I20240503 02:12:21.098162 29566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.102427 29566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077: Bootstrap starting.
I20240503 02:12:21.103621 29566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.105083 29566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077: No bootstrap required, opened a new log
I20240503 02:12:21.105705 29566 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER }
I20240503 02:12:21.105847 29566 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.105942 29566 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d37cce7fba4462a6c9b1b060f68077, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.106153 29566 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER }
I20240503 02:12:21.106287 29566 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:21.106370 29566 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:21.106463 29566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.108233 29566 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER }
I20240503 02:12:21.108431 29566 leader_election.cc:302] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5d37cce7fba4462a6c9b1b060f68077; no voters:
I20240503 02:12:21.108793 29566 leader_election.cc:288] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:21.108906 29569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.109376 29569 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 1 LEADER]: Becoming Leader. State: Replica: a5d37cce7fba4462a6c9b1b060f68077, State: Running, Role: LEADER
I20240503 02:12:21.109675 29566 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:21.109629 29569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER }
I20240503 02:12:21.110500 29570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5d37cce7fba4462a6c9b1b060f68077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER } }
I20240503 02:12:21.110613 29571 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5d37cce7fba4462a6c9b1b060f68077. Latest consensus state: current_term: 1 leader_uuid: "a5d37cce7fba4462a6c9b1b060f68077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d37cce7fba4462a6c9b1b060f68077" member_type: VOTER } }
I20240503 02:12:21.110868 29571 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:21.110810 29570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:21.112076 29574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:21.113308 29574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:21.116943 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:12:21.117419 29574 catalog_manager.cc:1347] Generated new cluster ID: 5c3db26bff4642008492204b1f07b7c5
I20240503 02:12:21.117512 29574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:21.141163 29574 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:21.141971 29574 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:21.161361 29574 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077: Generated new TSK 0
I20240503 02:12:21.161763 29574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:21.182032 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:21.185593 29588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.186573 29589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.187435 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.188401 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:21.188627 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:21.188695 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:21.188752 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702341188734 us; error 0 us; skew 500 ppm
I20240503 02:12:21.188917 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:21.190162 27337 webserver.cc:460] Webserver started at http://127.26.178.65:33307/ using document root <none> and password file <none>
I20240503 02:12:21.190425 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:21.190507 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:21.190701 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:21.191380 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "55e4cfde56e2427ba0af966ceefc1bbb"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.193807 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:21.195371 29596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.195732 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:21.195860 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "55e4cfde56e2427ba0af966ceefc1bbb"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.195973 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:21.243036 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:21.243688 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:21.244257 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:21.245178 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:21.245257 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.245362 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:21.245422 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.258718 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:42169
I20240503 02:12:21.258755 29658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:42169 every 8 connection(s)
I20240503 02:12:21.260429 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:21.264237 29663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.265219 29664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.266250 27337 server_base.cc:1034] running on GCE node
W20240503 02:12:21.266999 29666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.267414 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:21.267519 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:21.267587 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702341267563 us; error 0 us; skew 500 ppm
I20240503 02:12:21.267778 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:21.268721 29659 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37987
I20240503 02:12:21.268877 29659 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:21.269335 29659 heartbeater.cc:511] Master 127.26.178.126:37987 requested a full tablet report, sending...
I20240503 02:12:21.269385 27337 webserver.cc:460] Webserver started at http://127.26.178.66:44889/ using document root <none> and password file <none>
I20240503 02:12:21.269623 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:21.269727 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:21.269878 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:21.270272 29530 ts_manager.cc:194] Registered new tserver with Master: 55e4cfde56e2427ba0af966ceefc1bbb (127.26.178.65:42169)
I20240503 02:12:21.270519 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "33099bb735e942f186fe30c66d962a43"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.271378 29530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55320
I20240503 02:12:21.273056 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:21.274546 29671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.274943 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:12:21.275038 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "33099bb735e942f186fe30c66d962a43"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.275137 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:21.296521 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:21.297111 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:21.297629 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:21.298415 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:21.298480 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.298552 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:21.298594 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.310125 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:33861
I20240503 02:12:21.310171 29734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:33861 every 8 connection(s)
I20240503 02:12:21.311755 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:21.315784 29738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:21.316723 29739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.317662 27337 server_base.cc:1034] running on GCE node
W20240503 02:12:21.318254 29741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:21.318625 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:21.318745 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:21.318812 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702341318789 us; error 0 us; skew 500 ppm
I20240503 02:12:21.319015 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:21.319108 29735 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37987
I20240503 02:12:21.319273 29735 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:21.319654 29735 heartbeater.cc:511] Master 127.26.178.126:37987 requested a full tablet report, sending...
I20240503 02:12:21.320518 27337 webserver.cc:460] Webserver started at http://127.26.178.67:36701/ using document root <none> and password file <none>
I20240503 02:12:21.320724 29530 ts_manager.cc:194] Registered new tserver with Master: 33099bb735e942f186fe30c66d962a43 (127.26.178.66:33861)
I20240503 02:12:21.320787 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:21.320899 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:21.321035 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:21.321627 29530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55328
I20240503 02:12:21.321717 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "e41ec023d4f34cf39bb2f2412f195721"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.324328 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:21.325850 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.326201 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:21.326330 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "e41ec023d4f34cf39bb2f2412f195721"
format_stamp: "Formatted at 2024-05-03 02:12:21 on dist-test-slave-0cbc"
I20240503 02:12:21.326445 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:21.337226 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:21.337783 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:21.338289 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:21.339164 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:21.339242 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.339352 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:21.339416 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:21.353109 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:41403
I20240503 02:12:21.353128 29809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:41403 every 8 connection(s)
I20240503 02:12:21.359958 29810 heartbeater.cc:347] Connected to a master server at 127.26.178.126:37987
I20240503 02:12:21.360098 29810 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:21.360450 29810 heartbeater.cc:511] Master 127.26.178.126:37987 requested a full tablet report, sending...
I20240503 02:12:21.361269 29530 ts_manager.cc:194] Registered new tserver with Master: e41ec023d4f34cf39bb2f2412f195721 (127.26.178.67:41403)
I20240503 02:12:21.361961 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007938844s
I20240503 02:12:21.362023 29530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55334
I20240503 02:12:22.272624 29659 heartbeater.cc:503] Master 127.26.178.126:37987 was elected leader, sending a full tablet report...
I20240503 02:12:22.322953 29735 heartbeater.cc:503] Master 127.26.178.126:37987 was elected leader, sending a full tablet report...
I20240503 02:12:22.363299 29810 heartbeater.cc:503] Master 127.26.178.126:37987 was elected leader, sending a full tablet report...
I20240503 02:12:23.116940 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:25.117353 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:27.117807 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:29.118327 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:31.118826 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:33.119333 29586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:33.444854 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:12:33.459288 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:33.473582 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:12:33.476476 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:12:33.484210 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:33.496754 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:12:33.499433 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:12:33.506974 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:33.519704 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:12:33.522233 27337 master.cc:537] Master@127.26.178.126:37987 shutting down...
I20240503 02:12:33.529881 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:33.530119 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:33.530236 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P a5d37cce7fba4462a6c9b1b060f68077: stopping tablet replica
I20240503 02:12:33.543669 27337 master.cc:552] Master@127.26.178.126:37987 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12520 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 02:12:33.553046 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:38149
I20240503 02:12:33.553598 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:33.556634 29814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.557456 29815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.558193 29817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.559290 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:33.559597 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:33.559672 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:33.559729 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702353559709 us; error 0 us; skew 500 ppm
I20240503 02:12:33.559899 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:33.561367 27337 webserver.cc:460] Webserver started at http://127.26.178.126:44483/ using document root <none> and password file <none>
I20240503 02:12:33.561596 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:33.561678 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:33.561812 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:33.562481 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.565238 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:33.567445 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.567780 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20240503 02:12:33.567898 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.568025 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:33.596863 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:33.597489 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:33.610800 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:38149
I20240503 02:12:33.610832 29873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:38149 every 8 connection(s)
I20240503 02:12:33.612303 29874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.616797 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f: Bootstrap starting.
I20240503 02:12:33.618220 29874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.620103 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f: No bootstrap required, opened a new log
I20240503 02:12:33.620855 29874 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER }
I20240503 02:12:33.621026 29874 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.621114 29874 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa485a5d4fd647a38b1a7a31fc8a3c0f, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.621348 29874 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER }
I20240503 02:12:33.621495 29874 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:33.621584 29874 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:33.621688 29874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.623626 29874 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER }
I20240503 02:12:33.623853 29874 leader_election.cc:302] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa485a5d4fd647a38b1a7a31fc8a3c0f; no voters:
I20240503 02:12:33.624310 29874 leader_election.cc:288] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:33.624467 29877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.624944 29877 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 1 LEADER]: Becoming Leader. State: Replica: aa485a5d4fd647a38b1a7a31fc8a3c0f, State: Running, Role: LEADER
I20240503 02:12:33.625221 29877 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER }
I20240503 02:12:33.625283 29874 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:33.625907 29879 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa485a5d4fd647a38b1a7a31fc8a3c0f. Latest consensus state: current_term: 1 leader_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER } }
I20240503 02:12:33.626071 29878 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa485a5d4fd647a38b1a7a31fc8a3c0f" member_type: VOTER } }
I20240503 02:12:33.626199 29879 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:33.626273 29878 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:33.630715 29881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:33.630745 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:12:33.632072 29881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:33.634950 29881 catalog_manager.cc:1347] Generated new cluster ID: f285abed56804e92ba5b7da14788229d
I20240503 02:12:33.635042 29881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:33.643373 29881 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:33.644186 29881 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:33.653174 29881 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f: Generated new TSK 0
I20240503 02:12:33.653506 29881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:33.663750 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:33.667644 29895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.668418 29896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.669474 29898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.670331 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:33.670761 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:33.670859 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:33.670918 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702353670899 us; error 0 us; skew 500 ppm
I20240503 02:12:33.671088 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:33.672631 27337 webserver.cc:460] Webserver started at http://127.26.178.65:34929/ using document root <none> and password file <none>
I20240503 02:12:33.672936 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:33.673028 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:33.673173 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:33.673844 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "b8c9e41789fe4b57871fb4c3af686a0f"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.676779 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:33.678485 29903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.678869 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:12:33.678993 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "b8c9e41789fe4b57871fb4c3af686a0f"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.679114 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:33.696517 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:33.697152 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:33.697790 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:33.698792 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:33.698879 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.698995 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:33.699050 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.712841 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:39983
I20240503 02:12:33.712867 29966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:39983 every 8 connection(s)
I20240503 02:12:33.714706 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:33.718681 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.719497 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.721204 27337 server_base.cc:1034] running on GCE node
W20240503 02:12:33.722122 29975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.722512 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:33.722594 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:33.722687 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702353722636 us; error 0 us; skew 500 ppm
I20240503 02:12:33.722856 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:33.724350 27337 webserver.cc:460] Webserver started at http://127.26.178.66:43665/ using document root <none> and password file <none>
I20240503 02:12:33.724506 29967 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38149
I20240503 02:12:33.724663 29967 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:33.724661 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:33.724798 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:33.724980 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:33.725183 29967 heartbeater.cc:511] Master 127.26.178.126:38149 requested a full tablet report, sending...
I20240503 02:12:33.725808 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "3d7ecb553f4c49e5970baf81807ed61c"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.726258 29839 ts_manager.cc:194] Registered new tserver with Master: b8c9e41789fe4b57871fb4c3af686a0f (127.26.178.65:39983)
I20240503 02:12:33.727880 29839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49336
I20240503 02:12:33.729233 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:33.730880 29979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.731210 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:33.731350 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "3d7ecb553f4c49e5970baf81807ed61c"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.731460 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:33.739178 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:33.739744 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:33.740283 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:33.741119 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:33.741185 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.741254 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:33.741305 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.753505 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:39539
I20240503 02:12:33.753540 30041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:39539 every 8 connection(s)
I20240503 02:12:33.755129 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:33.759021 30045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.759860 30046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.760761 27337 server_base.cc:1034] running on GCE node
W20240503 02:12:33.761701 30048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.762159 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:12:33.762276 30042 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38149
W20240503 02:12:33.762352 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:33.762434 30042 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:33.762485 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702353762451 us; error 0 us; skew 500 ppm
I20240503 02:12:33.762663 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:33.762883 30042 heartbeater.cc:511] Master 127.26.178.126:38149 requested a full tablet report, sending...
I20240503 02:12:33.763979 29839 ts_manager.cc:194] Registered new tserver with Master: 3d7ecb553f4c49e5970baf81807ed61c (127.26.178.66:39539)
I20240503 02:12:33.764353 27337 webserver.cc:460] Webserver started at http://127.26.178.67:46707/ using document root <none> and password file <none>
I20240503 02:12:33.764593 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:33.764683 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:33.764818 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:33.764973 29839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49340
I20240503 02:12:33.765739 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "a35f9ec501a84708926c1cd1c5471865"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.768249 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 02:12:33.769979 30053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.770365 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:12:33.770478 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "a35f9ec501a84708926c1cd1c5471865"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.770593 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:33.790498 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:33.791149 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:33.791720 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:33.792567 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:33.792636 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.792735 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:33.792788 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.807085 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:44549
I20240503 02:12:33.807118 30116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:44549 every 8 connection(s)
I20240503 02:12:33.808785 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:33.812763 30120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.813515 30123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:33.814520 30121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:33.815488 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:33.815795 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:33.815882 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:33.815891 30117 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38149
I20240503 02:12:33.815953 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702353815928 us; error 0 us; skew 500 ppm
I20240503 02:12:33.816062 30117 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:33.816107 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:33.816483 30117 heartbeater.cc:511] Master 127.26.178.126:38149 requested a full tablet report, sending...
I20240503 02:12:33.817471 29839 ts_manager.cc:194] Registered new tserver with Master: a35f9ec501a84708926c1cd1c5471865 (127.26.178.67:44549)
I20240503 02:12:33.817754 27337 webserver.cc:460] Webserver started at http://127.26.178.68:43859/ using document root <none> and password file <none>
I20240503 02:12:33.817994 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:33.818086 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:33.818220 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:33.818398 29839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49342
I20240503 02:12:33.818946 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-3-root/instance:
uuid: "cc2182d03cf44402ae18e6adf6e9cb18"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.821449 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:33.823261 30128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.823624 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:12:33.823721 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-3-root
uuid: "cc2182d03cf44402ae18e6adf6e9cb18"
format_stamp: "Formatted at 2024-05-03 02:12:33 on dist-test-slave-0cbc"
I20240503 02:12:33.823835 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714702282772525-27337-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:33.851048 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:33.851652 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:33.852193 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:33.853082 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:33.853147 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.853237 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:33.853291 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:33.866693 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.68:37867
I20240503 02:12:33.866741 30190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.68:37867 every 8 connection(s)
I20240503 02:12:33.873701 30191 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38149
I20240503 02:12:33.873863 30191 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:33.874239 30191 heartbeater.cc:511] Master 127.26.178.126:38149 requested a full tablet report, sending...
I20240503 02:12:33.875128 29839 ts_manager.cc:194] Registered new tserver with Master: cc2182d03cf44402ae18e6adf6e9cb18 (127.26.178.68:37867)
I20240503 02:12:33.875679 27337 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.008059024s
I20240503 02:12:33.876096 29839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49350
I20240503 02:12:34.729349 29967 heartbeater.cc:503] Master 127.26.178.126:38149 was elected leader, sending a full tablet report...
I20240503 02:12:34.766332 30042 heartbeater.cc:503] Master 127.26.178.126:38149 was elected leader, sending a full tablet report...
I20240503 02:12:34.819744 30117 heartbeater.cc:503] Master 127.26.178.126:38149 was elected leader, sending a full tablet report...
I20240503 02:12:34.877507 30191 heartbeater.cc:503] Master 127.26.178.126:38149 was elected leader, sending a full tablet report...
I20240503 02:12:34.912034 27337 test_util.cc:268] Using random seed: 1086493466
I20240503 02:12:34.923032 29839 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49372:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:12:34.954455 30156 tablet_service.cc:1467] Processing CreateTablet for tablet e5bab1e491114db8a2b4fe0a87a4cb66 (DEFAULT_TABLE table=test-workload [id=64d7c5dc7e264fb4924e783dc9d35f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.955020 30156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bab1e491114db8a2b4fe0a87a4cb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.957975 30082 tablet_service.cc:1467] Processing CreateTablet for tablet e5bab1e491114db8a2b4fe0a87a4cb66 (DEFAULT_TABLE table=test-workload [id=64d7c5dc7e264fb4924e783dc9d35f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.958382 30082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bab1e491114db8a2b4fe0a87a4cb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.959048 30007 tablet_service.cc:1467] Processing CreateTablet for tablet e5bab1e491114db8a2b4fe0a87a4cb66 (DEFAULT_TABLE table=test-workload [id=64d7c5dc7e264fb4924e783dc9d35f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.959396 30007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bab1e491114db8a2b4fe0a87a4cb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.963420 30212 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: Bootstrap starting.
I20240503 02:12:34.964025 30213 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: Bootstrap starting.
I20240503 02:12:34.965121 30212 tablet_bootstrap.cc:654] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.965869 30213 tablet_bootstrap.cc:654] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.973536 30214 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: Bootstrap starting.
I20240503 02:12:34.973827 30213 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: No bootstrap required, opened a new log
I20240503 02:12:34.973824 30212 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: No bootstrap required, opened a new log
I20240503 02:12:34.973987 30213 ts_tablet_manager.cc:1367] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:12:34.973994 30212 ts_tablet_manager.cc:1367] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:12:34.974946 30213 raft_consensus.cc:348] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.974989 30212 raft_consensus.cc:348] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.975112 30213 raft_consensus.cc:374] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.975162 30212 raft_consensus.cc:374] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.975172 30213 raft_consensus.cc:729] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7ecb553f4c49e5970baf81807ed61c, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.975278 30212 raft_consensus.cc:729] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2182d03cf44402ae18e6adf6e9cb18, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.975364 30214 tablet_bootstrap.cc:654] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.975607 30212 consensus_queue.cc:260] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.975677 30213 consensus_queue.cc:260] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.976900 30212 ts_tablet_manager.cc:1398] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.979746 30213 ts_tablet_manager.cc:1398] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:12:34.980125 30214 tablet_bootstrap.cc:492] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: No bootstrap required, opened a new log
I20240503 02:12:34.981320 30214 ts_tablet_manager.cc:1367] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:12:34.982069 30214 raft_consensus.cc:348] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.982223 30214 raft_consensus.cc:374] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.982321 30214 raft_consensus.cc:729] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35f9ec501a84708926c1cd1c5471865, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.982558 30214 consensus_queue.cc:260] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:34.983247 30214 ts_tablet_manager.cc:1398] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 02:12:35.005803 30043 tablet.cc:2369] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:35.054260 30217 raft_consensus.cc:482] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.054461 30217 raft_consensus.cc:504] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:35.055613 30217 leader_election.cc:288] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d7ecb553f4c49e5970baf81807ed61c (127.26.178.66:39539), a35f9ec501a84708926c1cd1c5471865 (127.26.178.67:44549)
I20240503 02:12:35.061524 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bab1e491114db8a2b4fe0a87a4cb66" candidate_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7ecb553f4c49e5970baf81807ed61c" is_pre_election: true
I20240503 02:12:35.061524 30092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bab1e491114db8a2b4fe0a87a4cb66" candidate_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35f9ec501a84708926c1cd1c5471865" is_pre_election: true
I20240503 02:12:35.061838 30092 raft_consensus.cc:2454] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2182d03cf44402ae18e6adf6e9cb18 in term 0.
I20240503 02:12:35.061838 30017 raft_consensus.cc:2454] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2182d03cf44402ae18e6adf6e9cb18 in term 0.
I20240503 02:12:35.062299 30131 leader_election.cc:302] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7ecb553f4c49e5970baf81807ed61c, cc2182d03cf44402ae18e6adf6e9cb18; no voters:
I20240503 02:12:35.062566 30217 raft_consensus.cc:2781] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.062642 30217 raft_consensus.cc:482] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.062723 30217 raft_consensus.cc:3037] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.064512 30217 raft_consensus.cc:504] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:35.065034 30217 leader_election.cc:288] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [CANDIDATE]: Term 1 election: Requested vote from peers 3d7ecb553f4c49e5970baf81807ed61c (127.26.178.66:39539), a35f9ec501a84708926c1cd1c5471865 (127.26.178.67:44549)
I20240503 02:12:35.065470 30092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bab1e491114db8a2b4fe0a87a4cb66" candidate_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35f9ec501a84708926c1cd1c5471865"
I20240503 02:12:35.065474 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bab1e491114db8a2b4fe0a87a4cb66" candidate_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7ecb553f4c49e5970baf81807ed61c"
I20240503 02:12:35.065717 30092 raft_consensus.cc:3037] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.065748 30017 raft_consensus.cc:3037] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.068588 30017 raft_consensus.cc:2454] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2182d03cf44402ae18e6adf6e9cb18 in term 1.
I20240503 02:12:35.068588 30092 raft_consensus.cc:2454] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2182d03cf44402ae18e6adf6e9cb18 in term 1.
I20240503 02:12:35.069048 30130 leader_election.cc:302] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d7ecb553f4c49e5970baf81807ed61c, a35f9ec501a84708926c1cd1c5471865, cc2182d03cf44402ae18e6adf6e9cb18; no voters:
I20240503 02:12:35.069370 30217 raft_consensus.cc:2781] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.069738 30217 raft_consensus.cc:686] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 1 LEADER]: Becoming Leader. State: Replica: cc2182d03cf44402ae18e6adf6e9cb18, State: Running, Role: LEADER
I20240503 02:12:35.069958 30217 consensus_queue.cc:237] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } }
I20240503 02:12:35.072696 29838 catalog_manager.cc:5484] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc2182d03cf44402ae18e6adf6e9cb18 (127.26.178.68). New cstate: current_term: 1 leader_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc2182d03cf44402ae18e6adf6e9cb18" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 37867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:35.580701 30225 consensus_queue.cc:1035] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7ecb553f4c49e5970baf81807ed61c" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.586010 30230 consensus_queue.cc:1035] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35f9ec501a84708926c1cd1c5471865" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.630318 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:35.630837 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:35.630959 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:37.631424 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:37.631858 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:37.631942 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:39.632315 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:39.632848 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:39.632949 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:41.633384 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:41.633895 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:41.634022 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:43.634469 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:43.635033 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:43.635143 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:45.635519 29892 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:45.635974 29892 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:45.636065 29892 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:46.136444 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:12:46.153093 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:46.167390 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:12:46.170266 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:12:46.179265 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:46.179528 27337 tablet_replica.cc:330] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c: stopping tablet replica
I20240503 02:12:46.179713 27337 raft_consensus.cc:2229] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.179890 27337 raft_consensus.cc:2258] T e5bab1e491114db8a2b4fe0a87a4cb66 P 3d7ecb553f4c49e5970baf81807ed61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.193951 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:12:46.196914 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:12:46.205978 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:46.206214 27337 tablet_replica.cc:330] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865: stopping tablet replica
I20240503 02:12:46.206408 27337 raft_consensus.cc:2229] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:46.206595 27337 raft_consensus.cc:2258] T e5bab1e491114db8a2b4fe0a87a4cb66 P a35f9ec501a84708926c1cd1c5471865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.220432 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:12:46.223534 27337 tablet_server.cc:178] TabletServer@127.26.178.68:0 shutting down...
I20240503 02:12:46.232244 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:46.232539 27337 tablet_replica.cc:330] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18: stopping tablet replica
I20240503 02:12:46.232724 27337 raft_consensus.cc:2229] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.233052 27337 raft_consensus.cc:2258] T e5bab1e491114db8a2b4fe0a87a4cb66 P cc2182d03cf44402ae18e6adf6e9cb18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.246865 27337 tablet_server.cc:194] TabletServer@127.26.178.68:0 shutdown complete.
I20240503 02:12:46.249899 27337 master.cc:537] Master@127.26.178.126:38149 shutting down...
I20240503 02:12:46.257952 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:46.258181 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:46.258262 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa485a5d4fd647a38b1a7a31fc8a3c0f: stopping tablet replica
I20240503 02:12:46.281124 27337 master.cc:552] Master@127.26.178.126:38149 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12741 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 02:12:46.295066 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:45453
I20240503 02:12:46.295562 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:46.298756 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.299618 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.300415 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:46.301396 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:46.301632 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:46.301683 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:46.301707 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702366301704 us; error 0 us; skew 500 ppm
I20240503 02:12:46.301820 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:46.311528 27337 webserver.cc:460] Webserver started at http://127.26.178.126:37253/ using document root <none> and password file <none>
I20240503 02:12:46.311787 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:46.311846 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:46.311942 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:46.312600 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "69049ac986cd4ee594f3b8b0d114a37d"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.315310 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:12:46.317220 30245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.317633 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:12:46.317734 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "69049ac986cd4ee594f3b8b0d114a37d"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.317842 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:46.332464 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:46.332976 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:46.344439 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:45453
I20240503 02:12:46.344452 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:45453 every 8 connection(s)
I20240503 02:12:46.345831 30298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.350060 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d: Bootstrap starting.
I20240503 02:12:46.351302 30298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.352892 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d: No bootstrap required, opened a new log
I20240503 02:12:46.353477 30298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER }
I20240503 02:12:46.353608 30298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.353648 30298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69049ac986cd4ee594f3b8b0d114a37d, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.353801 30298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER }
I20240503 02:12:46.353893 30298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:46.353929 30298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:46.353979 30298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.355868 30298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER }
I20240503 02:12:46.356060 30298 leader_election.cc:302] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69049ac986cd4ee594f3b8b0d114a37d; no voters:
I20240503 02:12:46.356460 30298 leader_election.cc:288] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:46.356621 30301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.357091 30301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 1 LEADER]: Becoming Leader. State: Replica: 69049ac986cd4ee594f3b8b0d114a37d, State: Running, Role: LEADER
I20240503 02:12:46.357302 30301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER }
I20240503 02:12:46.357307 30298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:46.358116 30302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69049ac986cd4ee594f3b8b0d114a37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER } }
I20240503 02:12:46.358067 30303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69049ac986cd4ee594f3b8b0d114a37d. Latest consensus state: current_term: 1 leader_uuid: "69049ac986cd4ee594f3b8b0d114a37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69049ac986cd4ee594f3b8b0d114a37d" member_type: VOTER } }
I20240503 02:12:46.358265 30302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:46.358327 30303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:46.363523 30306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:46.365751 30306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:46.366154 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:12:46.369273 30306 catalog_manager.cc:1347] Generated new cluster ID: f7921f6b5364426da48503bea04f2841
I20240503 02:12:46.369387 30306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:46.386336 30306 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:46.387256 30306 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:46.395399 30306 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d: Generated new TSK 0
I20240503 02:12:46.395689 30306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:46.399225 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:46.403021 30319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.403856 30320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.404712 30322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:46.405885 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:46.406131 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:46.406186 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:46.406215 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702366406211 us; error 0 us; skew 500 ppm
I20240503 02:12:46.406365 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:46.407842 27337 webserver.cc:460] Webserver started at http://127.26.178.65:34893/ using document root <none> and password file <none>
I20240503 02:12:46.408078 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:46.408141 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:46.408252 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:46.408943 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "65e5592fdf4a46b19c3ec570b20d9dfc"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.411787 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:46.413563 30327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.413983 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:46.414079 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "65e5592fdf4a46b19c3ec570b20d9dfc"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.414183 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:46.427294 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:46.427827 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:46.428373 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:46.429278 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:46.429333 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.429388 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:46.429417 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.441308 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:45543
I20240503 02:12:46.441324 30389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:45543 every 8 connection(s)
I20240503 02:12:46.443032 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:46.446724 30394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.447434 30395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.448307 30397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:46.449301 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:46.449273 30390 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45453
I20240503 02:12:46.449538 30390 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:46.449584 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:46.449648 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:46.449687 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702366449683 us; error 0 us; skew 500 ppm
I20240503 02:12:46.449831 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:46.449942 30390 heartbeater.cc:511] Master 127.26.178.126:45453 requested a full tablet report, sending...
I20240503 02:12:46.450956 30262 ts_manager.cc:194] Registered new tserver with Master: 65e5592fdf4a46b19c3ec570b20d9dfc (127.26.178.65:45543)
I20240503 02:12:46.451335 27337 webserver.cc:460] Webserver started at http://127.26.178.66:40639/ using document root <none> and password file <none>
I20240503 02:12:46.451562 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:46.451623 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:46.451743 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:46.452175 30262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50702
I20240503 02:12:46.452407 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "9f53bcdfc141477a95c60a6598861258"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.455224 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:12:46.456956 30402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.457422 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:12:46.457551 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "9f53bcdfc141477a95c60a6598861258"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.457688 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:46.483067 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:46.483615 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:46.484189 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:46.485225 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:46.485291 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.485373 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:46.485409 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.500041 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:40581
I20240503 02:12:46.500058 30465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:40581 every 8 connection(s)
I20240503 02:12:46.501590 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:46.505275 30469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.506150 30470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:46.506932 30472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:46.507519 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:46.507869 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:46.507952 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:46.508033 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702366508005 us; error 0 us; skew 500 ppm
I20240503 02:12:46.508224 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:46.508617 30466 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45453
I20240503 02:12:46.508813 30466 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:46.509272 30466 heartbeater.cc:511] Master 127.26.178.126:45453 requested a full tablet report, sending...
I20240503 02:12:46.509629 27337 webserver.cc:460] Webserver started at http://127.26.178.67:35133/ using document root <none> and password file <none>
I20240503 02:12:46.509883 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:46.509974 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:46.510120 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:46.510375 30262 ts_manager.cc:194] Registered new tserver with Master: 9f53bcdfc141477a95c60a6598861258 (127.26.178.66:40581)
I20240503 02:12:46.510983 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "6566ad4e25874aa490360cd672f04d5f"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.511500 30262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50708
I20240503 02:12:46.513868 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:46.515574 30477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.516026 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:46.516148 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "6566ad4e25874aa490360cd672f04d5f"
format_stamp: "Formatted at 2024-05-03 02:12:46 on dist-test-slave-0cbc"
I20240503 02:12:46.516273 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:46.534548 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:46.535180 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:46.535787 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:46.536712 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:46.536790 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.536900 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:46.536960 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:46.548501 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:42735
I20240503 02:12:46.548573 30539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:42735 every 8 connection(s)
I20240503 02:12:46.555073 30540 heartbeater.cc:347] Connected to a master server at 127.26.178.126:45453
I20240503 02:12:46.555229 30540 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:46.555621 30540 heartbeater.cc:511] Master 127.26.178.126:45453 requested a full tablet report, sending...
I20240503 02:12:46.556500 30262 ts_manager.cc:194] Registered new tserver with Master: 6566ad4e25874aa490360cd672f04d5f (127.26.178.67:42735)
I20240503 02:12:46.557233 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007908017s
I20240503 02:12:46.557417 30262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50724
I20240503 02:12:47.453522 30390 heartbeater.cc:503] Master 127.26.178.126:45453 was elected leader, sending a full tablet report...
I20240503 02:12:47.513100 30466 heartbeater.cc:503] Master 127.26.178.126:45453 was elected leader, sending a full tablet report...
I20240503 02:12:47.558862 30540 heartbeater.cc:503] Master 127.26.178.126:45453 was elected leader, sending a full tablet report...
I20240503 02:12:47.592849 27337 test_util.cc:268] Using random seed: 1099174269
I20240503 02:12:47.604521 30262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39296:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:12:47.605072 30262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:12:47.625636 30355 tablet_service.cc:1467] Processing CreateTablet for tablet 72edfd79863445a5906735c6ba6165dc (DEFAULT_TABLE table=test-workload [id=b12af55b80fc4655b3bb6a96461be3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.626199 30355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72edfd79863445a5906735c6ba6165dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.630497 30505 tablet_service.cc:1467] Processing CreateTablet for tablet 72edfd79863445a5906735c6ba6165dc (DEFAULT_TABLE table=test-workload [id=b12af55b80fc4655b3bb6a96461be3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.630753 30430 tablet_service.cc:1467] Processing CreateTablet for tablet 72edfd79863445a5906735c6ba6165dc (DEFAULT_TABLE table=test-workload [id=b12af55b80fc4655b3bb6a96461be3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.630988 30505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72edfd79863445a5906735c6ba6165dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.631134 30430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72edfd79863445a5906735c6ba6165dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.632424 30562 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: Bootstrap starting.
I20240503 02:12:47.634184 30562 tablet_bootstrap.cc:654] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.636224 30562 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: No bootstrap required, opened a new log
I20240503 02:12:47.636384 30562 ts_tablet_manager.cc:1367] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:47.637246 30564 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: Bootstrap starting.
I20240503 02:12:47.637238 30562 raft_consensus.cc:348] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.637596 30562 raft_consensus.cc:374] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.637717 30562 raft_consensus.cc:729] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e5592fdf4a46b19c3ec570b20d9dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.638037 30562 consensus_queue.cc:260] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.639065 30564 tablet_bootstrap.cc:654] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.639120 30562 ts_tablet_manager.cc:1398] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:12:47.640935 30564 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: No bootstrap required, opened a new log
I20240503 02:12:47.641104 30564 ts_tablet_manager.cc:1367] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:47.641245 30565 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: Bootstrap starting.
I20240503 02:12:47.641969 30564 raft_consensus.cc:348] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.642190 30564 raft_consensus.cc:374] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.642293 30564 raft_consensus.cc:729] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f53bcdfc141477a95c60a6598861258, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.642590 30564 consensus_queue.cc:260] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.642858 30565 tablet_bootstrap.cc:654] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.643512 30564 ts_tablet_manager.cc:1398] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:47.645160 30565 tablet_bootstrap.cc:492] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: No bootstrap required, opened a new log
I20240503 02:12:47.645382 30565 ts_tablet_manager.cc:1367] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:47.646136 30565 raft_consensus.cc:348] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.646374 30565 raft_consensus.cc:374] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.646543 30565 raft_consensus.cc:729] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6566ad4e25874aa490360cd672f04d5f, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.646831 30565 consensus_queue.cc:260] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:47.647555 30565 ts_tablet_manager.cc:1398] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:48.058696 30568 raft_consensus.cc:482] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.058965 30568 raft_consensus.cc:504] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:48.060065 30568 leader_election.cc:288] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65e5592fdf4a46b19c3ec570b20d9dfc (127.26.178.65:45543), 6566ad4e25874aa490360cd672f04d5f (127.26.178.67:42735)
I20240503 02:12:48.067313 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72edfd79863445a5906735c6ba6165dc" candidate_uuid: "9f53bcdfc141477a95c60a6598861258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6566ad4e25874aa490360cd672f04d5f" is_pre_election: true
I20240503 02:12:48.067595 30515 raft_consensus.cc:2454] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f53bcdfc141477a95c60a6598861258 in term 0.
I20240503 02:12:48.068125 30404 leader_election.cc:302] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6566ad4e25874aa490360cd672f04d5f, 9f53bcdfc141477a95c60a6598861258; no voters:
I20240503 02:12:48.068284 30365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72edfd79863445a5906735c6ba6165dc" candidate_uuid: "9f53bcdfc141477a95c60a6598861258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" is_pre_election: true
I20240503 02:12:48.068460 30568 raft_consensus.cc:2781] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.068531 30365 raft_consensus.cc:2454] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f53bcdfc141477a95c60a6598861258 in term 0.
I20240503 02:12:48.068572 30568 raft_consensus.cc:482] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.068686 30568 raft_consensus.cc:3037] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.070603 30568 raft_consensus.cc:504] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:48.071213 30568 leader_election.cc:288] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [CANDIDATE]: Term 1 election: Requested vote from peers 65e5592fdf4a46b19c3ec570b20d9dfc (127.26.178.65:45543), 6566ad4e25874aa490360cd672f04d5f (127.26.178.67:42735)
I20240503 02:12:48.071672 30365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72edfd79863445a5906735c6ba6165dc" candidate_uuid: "9f53bcdfc141477a95c60a6598861258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc"
I20240503 02:12:48.071875 30365 raft_consensus.cc:3037] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.071849 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72edfd79863445a5906735c6ba6165dc" candidate_uuid: "9f53bcdfc141477a95c60a6598861258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6566ad4e25874aa490360cd672f04d5f"
I20240503 02:12:48.072070 30515 raft_consensus.cc:3037] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.074831 30365 raft_consensus.cc:2454] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f53bcdfc141477a95c60a6598861258 in term 1.
I20240503 02:12:48.074831 30515 raft_consensus.cc:2454] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f53bcdfc141477a95c60a6598861258 in term 1.
I20240503 02:12:48.075330 30404 leader_election.cc:302] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e5592fdf4a46b19c3ec570b20d9dfc, 9f53bcdfc141477a95c60a6598861258; no voters:
I20240503 02:12:48.075588 30568 raft_consensus.cc:2781] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.076025 30568 raft_consensus.cc:686] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 1 LEADER]: Becoming Leader. State: Replica: 9f53bcdfc141477a95c60a6598861258, State: Running, Role: LEADER
I20240503 02:12:48.076290 30568 consensus_queue.cc:237] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } }
I20240503 02:12:48.078820 30262 catalog_manager.cc:5484] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f53bcdfc141477a95c60a6598861258 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "9f53bcdfc141477a95c60a6598861258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f53bcdfc141477a95c60a6598861258" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 40581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:48.365631 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:48.366160 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:48.366626 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:48.366763 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:48.477295 30573 consensus_queue.cc:1035] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e5592fdf4a46b19c3ec570b20d9dfc" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.477645 30568 consensus_queue.cc:1035] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6566ad4e25874aa490360cd672f04d5f" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:49.368772 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:50.367240 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:50.367765 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:50.367882 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:50.371405 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:51.374269 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:52.368311 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:52.368774 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:52.368876 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:52.377223 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:53.379771 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:54.369387 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:54.369876 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:54.369985 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:54.383102 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:55.385957 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:56.370488 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:56.371052 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:56.371169 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:56.388724 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:57.391363 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:58.371699 30316 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:12:58.372139 30316 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:12:58.372247 30316 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:12:58.393975 30311 placement_policy_util.cc:407] tablet 72edfd79863445a5906735c6ba6165dc: detected majority of replicas (2 of 3) at location L0
I20240503 02:12:59.237545 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
I20240503 02:12:59.248335 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:59.248718 27337 tablet_replica.cc:330] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc: stopping tablet replica
I20240503 02:12:59.248899 27337 raft_consensus.cc:2229] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.249094 27337 raft_consensus.cc:2258] T 72edfd79863445a5906735c6ba6165dc P 65e5592fdf4a46b19c3ec570b20d9dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.263639 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:12:59.273798 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:12:59.282851 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:59.283142 27337 tablet_replica.cc:330] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258: stopping tablet replica
I20240503 02:12:59.283327 27337 raft_consensus.cc:2229] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.283677 27337 raft_consensus.cc:2258] T 72edfd79863445a5906735c6ba6165dc P 9f53bcdfc141477a95c60a6598861258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.298264 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:12:59.301524 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:12:59.310542 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:12:59.310889 27337 tablet_replica.cc:330] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f: stopping tablet replica
I20240503 02:12:59.311079 27337 raft_consensus.cc:2229] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.311287 27337 raft_consensus.cc:2258] T 72edfd79863445a5906735c6ba6165dc P 6566ad4e25874aa490360cd672f04d5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.325186 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:12:59.328536 27337 master.cc:537] Master@127.26.178.126:45453 shutting down...
I20240503 02:12:59.335944 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.336233 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.336361 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 69049ac986cd4ee594f3b8b0d114a37d: stopping tablet replica
I20240503 02:12:59.350047 27337 master.cc:552] Master@127.26.178.126:45453 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13066 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 02:12:59.361641 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:43777
I20240503 02:12:59.362203 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:59.366026 30586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.366725 30585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.367666 30588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:59.368683 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:59.369066 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:59.369129 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:59.369158 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702379369151 us; error 0 us; skew 500 ppm
I20240503 02:12:59.369376 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:59.376391 27337 webserver.cc:460] Webserver started at http://127.26.178.126:44287/ using document root <none> and password file <none>
I20240503 02:12:59.376678 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:59.376751 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:59.376901 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:59.377609 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "71239a2f32f14ce3921187b9bbcdf86f"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.380959 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 02:12:59.382870 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.383324 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:12:59.383451 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "71239a2f32f14ce3921187b9bbcdf86f"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.383584 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:59.399189 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:59.399803 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:59.411114 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:43777
I20240503 02:12:59.411130 30645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:43777 every 8 connection(s)
I20240503 02:12:59.412447 30646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.416702 30646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f: Bootstrap starting.
I20240503 02:12:59.418026 30646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.419816 30646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f: No bootstrap required, opened a new log
I20240503 02:12:59.420450 30646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER }
I20240503 02:12:59.420593 30646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.420627 30646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71239a2f32f14ce3921187b9bbcdf86f, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.420825 30646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER }
I20240503 02:12:59.420958 30646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:59.421017 30646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:59.421090 30646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.422978 30646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER }
I20240503 02:12:59.423185 30646 leader_election.cc:302] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71239a2f32f14ce3921187b9bbcdf86f; no voters:
I20240503 02:12:59.423607 30646 leader_election.cc:288] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:59.423743 30649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.424235 30649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 1 LEADER]: Becoming Leader. State: Replica: 71239a2f32f14ce3921187b9bbcdf86f, State: Running, Role: LEADER
I20240503 02:12:59.424600 30646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:59.424551 30649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER }
I20240503 02:12:59.425441 30650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71239a2f32f14ce3921187b9bbcdf86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER } }
I20240503 02:12:59.425499 30651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71239a2f32f14ce3921187b9bbcdf86f. Latest consensus state: current_term: 1 leader_uuid: "71239a2f32f14ce3921187b9bbcdf86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71239a2f32f14ce3921187b9bbcdf86f" member_type: VOTER } }
I20240503 02:12:59.425755 30651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:59.426275 30650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:59.426833 30655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:59.428073 30655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:59.431341 30655 catalog_manager.cc:1347] Generated new cluster ID: a5064396bd8a4e5b9a78f132cff0151f
I20240503 02:12:59.431466 30655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:59.431612 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:12:59.444360 30655 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:59.445168 30655 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:59.453487 30655 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f: Generated new TSK 0
I20240503 02:12:59.453744 30655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:59.464566 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:59.468233 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.469166 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.470166 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:59.471006 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:59.471318 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:59.471405 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:59.471432 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702379471429 us; error 0 us; skew 500 ppm
I20240503 02:12:59.471544 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:59.473019 27337 webserver.cc:460] Webserver started at http://127.26.178.65:42035/ using document root <none> and password file <none>
I20240503 02:12:59.473244 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:59.473309 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:59.473439 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:59.474061 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.476850 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:59.478639 30675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.479025 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:12:59.479130 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.479267 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:59.503863 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:59.504499 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:59.505139 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:59.506073 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:59.506134 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.506215 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:59.506253 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.519519 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:40107
I20240503 02:12:59.519536 30737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:40107 every 8 connection(s)
I20240503 02:12:59.521126 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:59.525256 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.526005 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.526487 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:59.527488 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:59.528100 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:59.528167 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:59.528195 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702379528190 us; error 0 us; skew 500 ppm
I20240503 02:12:59.528340 30738 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:12:59.528376 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:59.528457 30738 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:59.528831 30738 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:12:59.529814 27337 webserver.cc:460] Webserver started at http://127.26.178.66:33055/ using document root <none> and password file <none>
I20240503 02:12:59.529923 30610 ts_manager.cc:194] Registered new tserver with Master: ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:12:59.529995 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:59.530042 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:59.530133 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:59.530766 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.531108 30610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33264
I20240503 02:12:59.533447 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:59.535081 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.535442 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:12:59.535531 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.535626 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:59.548902 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:59.549471 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:59.550021 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:59.550947 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:59.551004 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.551056 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:59.551084 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.563244 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:46225
I20240503 02:12:59.563262 30813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:46225 every 8 connection(s)
I20240503 02:12:59.565008 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:59.568785 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.569658 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:59.570511 30820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:59.571185 30814 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:12:59.571321 30814 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:59.571344 27337 server_base.cc:1034] running on GCE node
I20240503 02:12:59.571674 30814 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:12:59.571792 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:12:59.571862 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:12:59.571903 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702379571896 us; error 0 us; skew 500 ppm
I20240503 02:12:59.572046 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:59.572635 30610 ts_manager.cc:194] Registered new tserver with Master: 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:12:59.573585 27337 webserver.cc:460] Webserver started at http://127.26.178.67:37499/ using document root <none> and password file <none>
I20240503 02:12:59.573720 30610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33266
I20240503 02:12:59.573858 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:59.573933 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:59.574052 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:59.574754 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.577315 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:59.578989 30825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.579344 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:12:59.579433 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
format_stamp: "Formatted at 2024-05-03 02:12:59 on dist-test-slave-0cbc"
I20240503 02:12:59.579545 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:59.619751 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:59.620306 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:59.620880 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:59.621768 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:59.621829 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.621906 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:59.621942 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:59.634387 30887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:44143 every 8 connection(s)
I20240503 02:12:59.634361 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:44143
I20240503 02:12:59.641242 30888 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:12:59.641397 30888 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:59.641745 30888 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:12:59.642776 30610 ts_manager.cc:194] Registered new tserver with Master: d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:12:59.643414 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007997155s
I20240503 02:12:59.643687 30610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33274
I20240503 02:13:00.532626 30738 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:00.575116 30814 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:00.645161 30888 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:00.671835 27337 test_util.cc:268] Using random seed: 1112253269
I20240503 02:13:00.685247 30610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33276:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:13:00.686115 30610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:00.717835 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d6429d7dd499fbe2957fb46fdf30b (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:13:00.717945 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 60cc6188939e4b728b1c1efea836a7bb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:13:00.718384 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d6429d7dd499fbe2957fb46fdf30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.718500 30851 tablet_service.cc:1467] Processing CreateTablet for tablet c2e4475d5b86469e90a10f84a910163a (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:13:00.718988 30850 tablet_service.cc:1467] Processing CreateTablet for tablet c8c3012068484fe3acd108cc36eb62d5 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:13:00.719039 30703 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d6429d7dd499fbe2957fb46fdf30b (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:13:00.719302 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec68f05310c4dcc8e02dadf0b4934ce (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:13:00.719456 30703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d6429d7dd499fbe2957fb46fdf30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.719420 30849 tablet_service.cc:1467] Processing CreateTablet for tablet d906796832b043e1ae7fe4825491a008 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:13:00.719679 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec68f05310c4dcc8e02dadf0b4934ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.719826 30848 tablet_service.cc:1467] Processing CreateTablet for tablet 5277bc5b18a0494baf73181fb4eeba01 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:13:00.720024 30694 tablet_service.cc:1467] Processing CreateTablet for tablet c81f82e2608a401ba8f4f381c3ad95be (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:13:00.720271 30847 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec68f05310c4dcc8e02dadf0b4934ce (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:13:00.720383 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81f82e2608a401ba8f4f381c3ad95be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.720739 30846 tablet_service.cc:1467] Processing CreateTablet for tablet 106b5308c1b34e119da27c3c75178ecb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:13:00.722379 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 5277bc5b18a0494baf73181fb4eeba01 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:13:00.722788 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5277bc5b18a0494baf73181fb4eeba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.723003 30696 tablet_service.cc:1467] Processing CreateTablet for tablet 106b5308c1b34e119da27c3c75178ecb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:13:00.723382 30696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b5308c1b34e119da27c3c75178ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.725620 30909 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.725864 30844 tablet_service.cc:1467] Processing CreateTablet for tablet c81f82e2608a401ba8f4f381c3ad95be (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:13:00.727164 30909 tablet_bootstrap.cc:654] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.727016 30779 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3d6429d7dd499fbe2957fb46fdf30b (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:13:00.727048 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a8cda34b54f4482afed13d9e43c4f (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:13:00.727290 30699 tablet_service.cc:1467] Processing CreateTablet for tablet d906796832b043e1ae7fe4825491a008 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:13:00.727473 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d6429d7dd499fbe2957fb46fdf30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.727473 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 60cc6188939e4b728b1c1efea836a7bb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:13:00.727766 30777 tablet_service.cc:1467] Processing CreateTablet for tablet c2e4475d5b86469e90a10f84a910163a (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:13:00.727877 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cc6188939e4b728b1c1efea836a7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.728137 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e4475d5b86469e90a10f84a910163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.728127 30700 tablet_service.cc:1467] Processing CreateTablet for tablet c8c3012068484fe3acd108cc36eb62d5 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:13:00.728430 30776 tablet_service.cc:1467] Processing CreateTablet for tablet c8c3012068484fe3acd108cc36eb62d5 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:13:00.728803 30701 tablet_service.cc:1467] Processing CreateTablet for tablet c2e4475d5b86469e90a10f84a910163a (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:13:00.729148 30775 tablet_service.cc:1467] Processing CreateTablet for tablet d906796832b043e1ae7fe4825491a008 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:13:00.719053 30845 tablet_service.cc:1467] Processing CreateTablet for tablet 441b8fdd54574e0f837d04e0916d1bf1 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:13:00.729838 30773 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec68f05310c4dcc8e02dadf0b4934ce (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:13:00.723016 30695 tablet_service.cc:1467] Processing CreateTablet for tablet 441b8fdd54574e0f837d04e0916d1bf1 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:13:00.727016 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 5277bc5b18a0494baf73181fb4eeba01 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:13:00.730871 30771 tablet_service.cc:1467] Processing CreateTablet for tablet 441b8fdd54574e0f837d04e0916d1bf1 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:13:00.731563 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e4475d5b86469e90a10f84a910163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.731588 30770 tablet_service.cc:1467] Processing CreateTablet for tablet c81f82e2608a401ba8f4f381c3ad95be (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:13:00.734848 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a8cda34b54f4482afed13d9e43c4f (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:13:00.720024 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 60cc6188939e4b728b1c1efea836a7bb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:13:00.728811 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c3012068484fe3acd108cc36eb62d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.727473 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a8cda34b54f4482afed13d9e43c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.737496 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c3012068484fe3acd108cc36eb62d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.739121 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec68f05310c4dcc8e02dadf0b4934ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.739425 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d906796832b043e1ae7fe4825491a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.730517 30772 tablet_service.cc:1467] Processing CreateTablet for tablet 106b5308c1b34e119da27c3c75178ecb (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:13:00.740988 30845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441b8fdd54574e0f837d04e0916d1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.743568 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c3012068484fe3acd108cc36eb62d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.743842 30701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e4475d5b86469e90a10f84a910163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.743908 30771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441b8fdd54574e0f837d04e0916d1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.745623 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cc6188939e4b728b1c1efea836a7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.749857 30703 tablet_service.cc:1467] Processing CreateTablet for tablet 3f15a775e24c46eda9ad8f5e8a537da4 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:13:00.750559 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cc6188939e4b728b1c1efea836a7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.750768 30847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec68f05310c4dcc8e02dadf0b4934ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.750918 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5277bc5b18a0494baf73181fb4eeba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.754923 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441b8fdd54574e0f837d04e0916d1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.754952 30848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5277bc5b18a0494baf73181fb4eeba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.756430 30703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f15a775e24c46eda9ad8f5e8a537da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.759958 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b5308c1b34e119da27c3c75178ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.759969 30846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b5308c1b34e119da27c3c75178ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.762538 30849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d906796832b043e1ae7fe4825491a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.765316 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d906796832b043e1ae7fe4825491a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.765563 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 3f15a775e24c46eda9ad8f5e8a537da4 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:13:00.778965 30844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81f82e2608a401ba8f4f381c3ad95be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.786752 30770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81f82e2608a401ba8f4f381c3ad95be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.793583 30848 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a8cda34b54f4482afed13d9e43c4f (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:13:00.794266 30848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a8cda34b54f4482afed13d9e43c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.796962 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 3f15a775e24c46eda9ad8f5e8a537da4 (DEFAULT_TABLE table=test-workload [id=87461bef0fd849ac915fd86a51cee96f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:13:00.800688 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f15a775e24c46eda9ad8f5e8a537da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.805001 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f15a775e24c46eda9ad8f5e8a537da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.805907 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a8cda34b54f4482afed13d9e43c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.871474 30909 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.871798 30909 ts_tablet_manager.cc:1367] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.146s user 0.028s sys 0.028s
I20240503 02:13:00.873682 30910 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.873724 30911 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.874255 30909 raft_consensus.cc:348] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.874574 30909 raft_consensus.cc:374] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.874720 30909 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.875003 30909 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.876282 30910 tablet_bootstrap.cc:654] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.876394 30911 tablet_bootstrap.cc:654] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.879257 30909 ts_tablet_manager.cc:1398] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:13:00.879606 30909 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.880097 30911 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.880235 30911 ts_tablet_manager.cc:1367] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:13:00.880244 30910 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.880402 30910 ts_tablet_manager.cc:1367] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:13:00.881008 30911 raft_consensus.cc:348] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.881206 30911 raft_consensus.cc:374] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.881083 30910 raft_consensus.cc:348] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:00.881266 30909 tablet_bootstrap.cc:654] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.881282 30911 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.881314 30910 raft_consensus.cc:374] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.881373 30910 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.881522 30911 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.881610 30910 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:00.885131 30911 ts_tablet_manager.cc:1398] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 02:13:00.885425 30911 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.885632 30910 ts_tablet_manager.cc:1398] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 02:13:00.887018 30911 tablet_bootstrap.cc:654] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.887012 30910 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.888293 30909 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.888432 30909 ts_tablet_manager.cc:1367] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 02:13:00.889256 30910 tablet_bootstrap.cc:654] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.889153 30909 raft_consensus.cc:348] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:00.889358 30909 raft_consensus.cc:374] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.889425 30909 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.889694 30909 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:00.890219 30909 ts_tablet_manager.cc:1398] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:00.890460 30909 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.891979 30909 tablet_bootstrap.cc:654] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.892871 30910 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.892930 30911 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.893149 30911 ts_tablet_manager.cc:1367] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:13:00.893173 30910 ts_tablet_manager.cc:1367] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:13:00.893968 30911 raft_consensus.cc:348] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.894176 30911 raft_consensus.cc:374] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.894239 30911 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.894054 30910 raft_consensus.cc:348] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.894466 30911 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.894567 30910 raft_consensus.cc:374] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.894798 30910 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.894997 30911 ts_tablet_manager.cc:1398] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.895251 30911 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.895237 30910 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.896797 30911 tablet_bootstrap.cc:654] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.897006 30910 ts_tablet_manager.cc:1398] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:00.897374 30910 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.899412 30910 tablet_bootstrap.cc:654] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.900353 30909 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.900516 30909 ts_tablet_manager.cc:1367] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240503 02:13:00.901484 30909 raft_consensus.cc:348] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.901755 30909 raft_consensus.cc:374] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.901809 30909 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.902180 30909 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.902865 30909 ts_tablet_manager.cc:1398] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:00.904420 30909 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.905002 30911 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.905150 30911 ts_tablet_manager.cc:1367] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 02:13:00.905859 30911 raft_consensus.cc:348] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.906069 30911 raft_consensus.cc:374] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.906141 30911 raft_consensus.cc:729] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.906428 30909 tablet_bootstrap.cc:654] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.906385 30911 consensus_queue.cc:260] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.907377 30911 ts_tablet_manager.cc:1398] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:00.907672 30911 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.908934 30909 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.909099 30909 ts_tablet_manager.cc:1367] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:00.909236 30911 tablet_bootstrap.cc:654] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.909974 30909 raft_consensus.cc:348] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.910230 30909 raft_consensus.cc:374] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.910290 30909 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.910589 30909 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.911836 30911 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.911976 30911 ts_tablet_manager.cc:1367] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:13:00.912024 30909 ts_tablet_manager.cc:1398] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:00.912359 30909 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.912683 30911 raft_consensus.cc:348] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.912875 30911 raft_consensus.cc:374] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.912933 30911 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.913204 30911 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.913893 30911 ts_tablet_manager.cc:1398] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:00.914149 30911 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.914214 30909 tablet_bootstrap.cc:654] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.915673 30911 tablet_bootstrap.cc:654] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.916479 30910 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.916724 30910 ts_tablet_manager.cc:1367] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20240503 02:13:00.917755 30910 raft_consensus.cc:348] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.918200 30910 raft_consensus.cc:374] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.918321 30910 raft_consensus.cc:729] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.918704 30910 consensus_queue.cc:260] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.919595 30909 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.919728 30909 ts_tablet_manager.cc:1367] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:13:00.920163 30910 ts_tablet_manager.cc:1398] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:00.920481 30910 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.920429 30909 raft_consensus.cc:348] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.920619 30909 raft_consensus.cc:374] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.920678 30909 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.920905 30909 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.921453 30909 ts_tablet_manager.cc:1398] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.921742 30909 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.922426 30910 tablet_bootstrap.cc:654] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.923189 30909 tablet_bootstrap.cc:654] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.928676 30909 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.928869 30909 ts_tablet_manager.cc:1367] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 02:13:00.929909 30909 raft_consensus.cc:348] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.930197 30909 raft_consensus.cc:374] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.930269 30909 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.930385 30911 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.930531 30911 ts_tablet_manager.cc:1367] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20240503 02:13:00.930552 30909 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.931339 30909 ts_tablet_manager.cc:1398] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.931308 30911 raft_consensus.cc:348] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.931514 30911 raft_consensus.cc:374] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.931589 30911 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.931661 30909 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.931785 30911 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.932080 30910 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.932242 30910 ts_tablet_manager.cc:1367] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 02:13:00.932956 30910 raft_consensus.cc:348] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.933231 30910 raft_consensus.cc:374] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.933228 30909 tablet_bootstrap.cc:654] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.933429 30910 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.933751 30910 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.934520 30911 ts_tablet_manager.cc:1398] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:00.935415 30911 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.935830 30910 ts_tablet_manager.cc:1398] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:00.936304 30910 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.937141 30911 tablet_bootstrap.cc:654] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.938663 30910 tablet_bootstrap.cc:654] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.941694 30909 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.941897 30909 ts_tablet_manager.cc:1367] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 02:13:00.942376 30911 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.942505 30911 ts_tablet_manager.cc:1367] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:13:00.942893 30909 raft_consensus.cc:348] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.943181 30909 raft_consensus.cc:374] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.943130 30911 raft_consensus.cc:348] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.943257 30909 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.943323 30911 raft_consensus.cc:374] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.943392 30911 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.943620 30911 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.943571 30909 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.944294 30909 ts_tablet_manager.cc:1398] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:00.944648 30909 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.945852 30911 ts_tablet_manager.cc:1398] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:00.946142 30911 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.947845 30911 tablet_bootstrap.cc:654] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.948231 30909 tablet_bootstrap.cc:654] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.950716 30910 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.950901 30910 ts_tablet_manager.cc:1367] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240503 02:13:00.951954 30910 raft_consensus.cc:348] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.952284 30910 raft_consensus.cc:374] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.952368 30910 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.952728 30911 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.952674 30910 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.952863 30911 ts_tablet_manager.cc:1367] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 02:13:00.953456 30910 ts_tablet_manager.cc:1398] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.953542 30911 raft_consensus.cc:348] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.953722 30910 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.953773 30911 raft_consensus.cc:374] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.953855 30909 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.953869 30911 raft_consensus.cc:729] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.954010 30909 ts_tablet_manager.cc:1367] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 02:13:00.954156 30911 consensus_queue.cc:260] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.954877 30911 ts_tablet_manager.cc:1398] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:00.954816 30909 raft_consensus.cc:348] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.955022 30909 raft_consensus.cc:374] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.955107 30909 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.955160 30911 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.955515 30910 tablet_bootstrap.cc:654] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.955358 30909 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.956894 30911 tablet_bootstrap.cc:654] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.958009 30909 ts_tablet_manager.cc:1398] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:00.958369 30909 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.960903 30909 tablet_bootstrap.cc:654] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.964054 30910 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.964287 30910 ts_tablet_manager.cc:1367] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:13:00.965298 30910 raft_consensus.cc:348] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.965633 30910 raft_consensus.cc:374] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.965746 30910 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.966095 30910 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.969352 30910 ts_tablet_manager.cc:1398] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:00.969816 30910 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.972924 30910 tablet_bootstrap.cc:654] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.973523 30909 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.973709 30909 ts_tablet_manager.cc:1367] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.003s
I20240503 02:13:00.974647 30909 raft_consensus.cc:348] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.974990 30909 raft_consensus.cc:374] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.975039 30911 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.975104 30909 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.975212 30911 ts_tablet_manager.cc:1367] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.012s
I20240503 02:13:00.975468 30909 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.975950 30911 raft_consensus.cc:348] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.976213 30911 raft_consensus.cc:374] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.976312 30909 ts_tablet_manager.cc:1398] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:13:00.976315 30911 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.976733 30909 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.976661 30911 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.979229 30909 tablet_bootstrap.cc:654] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.980460 30911 ts_tablet_manager.cc:1398] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:00.981249 30911 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.981930 30910 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.982178 30910 ts_tablet_manager.cc:1367] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 02:13:00.983178 30911 tablet_bootstrap.cc:654] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.983168 30910 raft_consensus.cc:348] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.983510 30910 raft_consensus.cc:374] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.983964 30910 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.984256 30910 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.984778 30910 ts_tablet_manager.cc:1398] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:00.985147 30910 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:00.985266 30909 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.985461 30909 ts_tablet_manager.cc:1367] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 02:13:00.986459 30909 raft_consensus.cc:348] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.986778 30909 raft_consensus.cc:374] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.986884 30909 raft_consensus.cc:729] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.986954 30910 tablet_bootstrap.cc:654] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.987241 30909 consensus_queue.cc:260] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.989018 30909 ts_tablet_manager.cc:1398] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:00.989423 30909 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.991783 30909 tablet_bootstrap.cc:654] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.994177 30909 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:00.994177 30911 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:00.994392 30909 ts_tablet_manager.cc:1367] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:00.994607 30911 ts_tablet_manager.cc:1367] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240503 02:13:00.995114 30909 raft_consensus.cc:348] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.995340 30909 raft_consensus.cc:374] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.995431 30909 raft_consensus.cc:729] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.995669 30909 consensus_queue.cc:260] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.995617 30911 raft_consensus.cc:348] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.995896 30911 raft_consensus.cc:374] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.996002 30911 raft_consensus.cc:729] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.996320 30909 ts_tablet_manager.cc:1398] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.996368 30911 consensus_queue.cc:260] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:00.997076 30911 ts_tablet_manager.cc:1398] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:00.996625 30909 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: Bootstrap starting.
I20240503 02:13:00.997517 30911 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:00.999051 30910 tablet_bootstrap.cc:492] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:00.999241 30910 ts_tablet_manager.cc:1367] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240503 02:13:00.999418 30911 tablet_bootstrap.cc:654] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.000172 30910 raft_consensus.cc:348] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.000437 30910 raft_consensus.cc:374] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.000481 30909 tablet_bootstrap.cc:654] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.000564 30910 raft_consensus.cc:729] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.000980 30910 consensus_queue.cc:260] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.002703 30910 ts_tablet_manager.cc:1398] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:01.003041 30910 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:01.005419 30910 tablet_bootstrap.cc:654] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.006871 30909 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: No bootstrap required, opened a new log
I20240503 02:13:01.007048 30909 ts_tablet_manager.cc:1367] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240503 02:13:01.007761 30909 raft_consensus.cc:348] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.007958 30909 raft_consensus.cc:374] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.008050 30909 raft_consensus.cc:729] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.008132 30910 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:01.008303 30910 ts_tablet_manager.cc:1367] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:13:01.008325 30909 consensus_queue.cc:260] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.008641 30911 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:01.008790 30911 ts_tablet_manager.cc:1367] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:13:01.008878 30909 ts_tablet_manager.cc:1398] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:01.009030 30910 raft_consensus.cc:348] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.009256 30910 raft_consensus.cc:374] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.009352 30910 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.009466 30911 raft_consensus.cc:348] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:01.009660 30911 raft_consensus.cc:374] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.009765 30911 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.009665 30910 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.009976 30911 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:01.010360 30910 ts_tablet_manager.cc:1398] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:01.010496 30911 ts_tablet_manager.cc:1398] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:01.010706 30910 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:01.010761 30911 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:01.012641 30911 tablet_bootstrap.cc:654] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.012645 30910 tablet_bootstrap.cc:654] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.015796 30910 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:01.015973 30910 ts_tablet_manager.cc:1367] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 02:13:01.016207 30911 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:01.016381 30911 ts_tablet_manager.cc:1367] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 02:13:01.016681 30910 raft_consensus.cc:348] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.016865 30910 raft_consensus.cc:374] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.016960 30910 raft_consensus.cc:729] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.017129 30911 raft_consensus.cc:348] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.017201 30910 consensus_queue.cc:260] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.017349 30911 raft_consensus.cc:374] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.017454 30911 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.017683 30911 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.017783 30910 ts_tablet_manager.cc:1398] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:01.018080 30910 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:01.018193 30911 ts_tablet_manager.cc:1398] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:01.018537 30911 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Bootstrap starting.
I20240503 02:13:01.020007 30910 tablet_bootstrap.cc:654] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.020146 30918 raft_consensus.cc:482] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.020370 30911 tablet_bootstrap.cc:654] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.020318 30918 raft_consensus.cc:504] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.021672 30918 leader_election.cc:288] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.022274 30910 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:01.022457 30910 ts_tablet_manager.cc:1367] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:01.023257 30910 raft_consensus.cc:348] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.023532 30910 raft_consensus.cc:374] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.023626 30910 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.023896 30910 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.024443 30910 ts_tablet_manager.cc:1398] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:01.024775 30910 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Bootstrap starting.
I20240503 02:13:01.027092 30910 tablet_bootstrap.cc:654] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.028203 30911 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: No bootstrap required, opened a new log
I20240503 02:13:01.028412 30911 ts_tablet_manager.cc:1367] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 02:13:01.029517 30911 raft_consensus.cc:348] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.029839 30911 raft_consensus.cc:374] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.029964 30911 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.030237 30911 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.030987 30911 ts_tablet_manager.cc:1398] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:01.031304 30918 raft_consensus.cc:482] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.031522 30918 raft_consensus.cc:504] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.032341 30918 leader_election.cc:288] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.037818 30910 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: No bootstrap required, opened a new log
I20240503 02:13:01.038035 30910 ts_tablet_manager.cc:1367] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240503 02:13:01.038941 30910 raft_consensus.cc:348] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.039201 30910 raft_consensus.cc:374] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.039312 30910 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.039610 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.039904 30863 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.040395 30752 leader_election.cc:302] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.040766 30918 raft_consensus.cc:2781] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.040432 30910 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.040885 30918 raft_consensus.cc:482] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.041014 30918 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.040916 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.041041 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.041848 30862 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.042196 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.043118 30712 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.043956 30752 leader_election.cc:302] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.044306 30713 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.045001 30913 raft_consensus.cc:482] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.045166 30913 raft_consensus.cc:504] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.046519 30913 leader_election.cc:288] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.050107 30913 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.050410 30913 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:01.051430 30913 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.053414 30918 raft_consensus.cc:504] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.055229 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.055541 30713 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.058351 30916 raft_consensus.cc:482] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.058524 30916 raft_consensus.cc:504] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.059738 30713 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.055238 30918 leader_election.cc:288] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.060684 30754 leader_election.cc:302] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.059738 30916 leader_election.cc:288] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.061067 30916 raft_consensus.cc:482] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.061254 30918 raft_consensus.cc:482] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.061210 30916 raft_consensus.cc:504] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.061553 30918 raft_consensus.cc:504] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.061923 30916 leader_election.cc:288] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.062372 30918 leader_election.cc:288] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.062592 30918 raft_consensus.cc:2781] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.062758 30918 raft_consensus.cc:482] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.062916 30918 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.066988 30918 raft_consensus.cc:504] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.068233 30953 raft_consensus.cc:2781] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.069854 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.069916 30953 raft_consensus.cc:686] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:01.070159 30713 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.070722 30754 leader_election.cc:302] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.070498 30953 consensus_queue.cc:237] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.070802 30918 leader_election.cc:288] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.071730 30918 raft_consensus.cc:2781] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.071769 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.068527 30910 ts_tablet_manager.cc:1398] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Time spent starting tablet: real 0.030s user 0.018s sys 0.010s
I20240503 02:13:01.072265 30712 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.072571 30918 raft_consensus.cc:482] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.072889 30918 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.077235 30712 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.077960 30754 leader_election.cc:302] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.081605 30918 raft_consensus.cc:504] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.083814 30953 raft_consensus.cc:2781] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.084602 30953 raft_consensus.cc:686] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:01.084865 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.085127 30712 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.085601 30918 leader_election.cc:288] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.090245 30712 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.090924 30754 leader_election.cc:302] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.091341 30918 raft_consensus.cc:2781] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.085125 30953 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.091668 30918 raft_consensus.cc:686] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:01.092231 30918 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
W20240503 02:13:01.107851 30752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.178.67:44143, user_credentials={real_user=slave}} blocked reactor thread for 65094.5us
I20240503 02:13:01.109020 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.109336 30862 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.114391 30862 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.115928 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.116269 30862 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 0.
I20240503 02:13:01.117328 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.117591 30862 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.121819 30913 raft_consensus.cc:482] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.122040 30913 raft_consensus.cc:504] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.122713 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.122911 30863 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.123003 30913 leader_election.cc:288] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.127332 30863 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.122061 30862 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 1.
I20240503 02:13:01.135051 30913 raft_consensus.cc:482] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.135250 30913 raft_consensus.cc:504] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.136067 30913 leader_election.cc:288] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.138454 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.138871 30861 raft_consensus.cc:2454] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.140192 30607 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.141304 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.141422 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.141624 30863 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.141731 30789 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.141328 30607 catalog_manager.cc:5484] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.142305 30677 leader_election.cc:302] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.142691 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.142915 30863 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.142621 30828 leader_election.cc:302] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.142616 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.143421 30788 raft_consensus.cc:2454] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.143457 30913 raft_consensus.cc:2781] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.143594 30913 raft_consensus.cc:482] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.143715 30913 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.143937 30828 leader_election.cc:302] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.144542 30916 raft_consensus.cc:2781] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.144662 30916 raft_consensus.cc:482] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.144757 30916 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.145243 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.145447 30862 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.147608 30913 raft_consensus.cc:504] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.148037 30916 raft_consensus.cc:504] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.149284 30916 leader_election.cc:288] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.149547 30916 raft_consensus.cc:482] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.149758 30916 raft_consensus.cc:504] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.151484 30913 leader_election.cc:288] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.151731 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.151978 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.152108 30787 raft_consensus.cc:2454] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.152220 30788 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.152293 30913 raft_consensus.cc:482] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.152472 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.152604 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.152813 30786 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.152985 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.152822 30789 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.153228 30785 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.152704 30913 raft_consensus.cc:504] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.153916 30678 leader_election.cc:302] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.154255 30678 leader_election.cc:302] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.154294 30913 leader_election.cc:288] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.154783 30913 raft_consensus.cc:2781] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.154947 30913 raft_consensus.cc:482] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.155162 30913 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.154930 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.156831 30785 raft_consensus.cc:2454] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.143604 30677 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.158303 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.143455 30958 raft_consensus.cc:2781] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.158579 30863 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.158632 30958 raft_consensus.cc:482] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.159035 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.159391 30712 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.159592 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.158877 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.159929 30913 raft_consensus.cc:504] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.160118 30710 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.160411 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.160641 30862 raft_consensus.cc:2379] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ebd1bb9e6d2f4384b781596ce1795b54 in current term 1: Already voted for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in this term.
I20240503 02:13:01.160753 30916 leader_election.cc:288] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.160820 30913 leader_election.cc:288] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.161048 30913 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.161280 30966 raft_consensus.cc:2781] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.161721 30958 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.161731 30713 raft_consensus.cc:2379] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in current term 1: Already voted for candidate ebd1bb9e6d2f4384b781596ce1795b54 in this term.
I20240503 02:13:01.161339 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.162132 30966 raft_consensus.cc:482] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.162283 30913 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.162374 30677 leader_election.cc:302] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.162771 30711 raft_consensus.cc:2379] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in current term 1: Already voted for candidate ebd1bb9e6d2f4384b781596ce1795b54 in this term.
I20240503 02:13:01.162773 30913 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.163139 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.163458 30785 raft_consensus.cc:2379] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in current term 1: Already voted for candidate ebd1bb9e6d2f4384b781596ce1795b54 in this term.
I20240503 02:13:01.163722 30829 leader_election.cc:302] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.164279 30965 raft_consensus.cc:2781] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.164417 30965 raft_consensus.cc:482] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.164474 30967 raft_consensus.cc:2781] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.164595 30965 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.164592 30678 leader_election.cc:302] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.164726 30967 raft_consensus.cc:482] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.164878 30966 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.165155 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.165580 30789 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.165867 30967 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.166888 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.167157 30862 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.169711 30913 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:01.170642 30829 leader_election.cc:302] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54
I20240503 02:13:01.171463 30916 raft_consensus.cc:2726] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:01.171734 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.171799 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.171943 30966 raft_consensus.cc:504] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.172180 30863 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.171074 30967 raft_consensus.cc:504] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.174010 30913 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.174742 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.175210 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.175441 30789 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.175179 30786 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.175168 30785 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.177017 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.178148 30861 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.178531 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.177845 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.177790 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.179828 30788 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.179872 30860 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.180058 30789 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.180644 30913 raft_consensus.cc:2781] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.182255 30967 leader_election.cc:288] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.183336 30966 leader_election.cc:288] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.183833 30913 raft_consensus.cc:686] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.184798 30958 raft_consensus.cc:504] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.186818 30863 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.186870 30862 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.187719 30677 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.188997 30677 leader_election.cc:302] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.189810 30966 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.188037 30913 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.191380 30967 raft_consensus.cc:2781] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.192235 30958 leader_election.cc:288] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.193782 30968 raft_consensus.cc:482] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.194029 30968 raft_consensus.cc:504] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.194825 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.196724 30966 raft_consensus.cc:686] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.196873 30968 leader_election.cc:288] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.197324 30966 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } }
I20240503 02:13:01.197746 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.201653 30711 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.202292 30787 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.205664 30965 raft_consensus.cc:504] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.207336 30785 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.208042 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" is_pre_election: true
I20240503 02:13:01.209270 30786 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.209566 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.211628 30713 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.211928 30863 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.212484 30677 leader_election.cc:302] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.212879 30968 raft_consensus.cc:2781] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.215128 30861 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.218607 30713 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.218984 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.219369 30968 raft_consensus.cc:482] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.220130 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.220433 30784 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 0.
I20240503 02:13:01.221175 30965 leader_election.cc:288] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.221448 30968 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.222532 30860 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.228200 30677 leader_election.cc:302] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.228876 30677 leader_election.cc:302] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.229900 30829 leader_election.cc:302] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.230296 30965 raft_consensus.cc:2781] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.230594 30965 raft_consensus.cc:686] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:01.230911 30965 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.240953 30787 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.241959 30828 leader_election.cc:302] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.242481 30958 raft_consensus.cc:2781] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.244189 30968 raft_consensus.cc:504] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.244887 30789 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.245631 30958 raft_consensus.cc:686] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:01.245632 30786 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.246397 30913 raft_consensus.cc:2781] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.247751 30958 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.248059 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:01.247967 30711 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.248217 30968 leader_election.cc:288] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:01.248510 30967 raft_consensus.cc:686] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.248865 30967 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.250958 30607 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.252264 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.252652 30913 raft_consensus.cc:686] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.252969 30913 consensus_queue.cc:237] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.253453 30860 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.253494 30970 raft_consensus.cc:2781] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.253664 30970 raft_consensus.cc:686] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.253950 30970 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.254751 30610 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:01.255774 30789 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.256999 30860 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.257495 30677 leader_election.cc:302] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.257889 30913 raft_consensus.cc:2781] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.258060 30913 raft_consensus.cc:686] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.258255 30789 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.258459 30913 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.263218 30788 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 1.
I20240503 02:13:01.265856 30786 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.268011 30610 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:01.269407 30607 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:01.269907 30607 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.270332 30607 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.270812 30607 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.271296 30607 catalog_manager.cc:5484] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.388978 30965 raft_consensus.cc:482] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.389170 30965 raft_consensus.cc:504] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.389952 30965 leader_election.cc:288] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.390534 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" is_pre_election: true
I20240503 02:13:01.390540 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" is_pre_election: true
I20240503 02:13:01.390893 30786 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.390873 30711 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 0.
I20240503 02:13:01.391320 30829 leader_election.cc:302] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:01.391593 30965 raft_consensus.cc:2781] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.391706 30965 raft_consensus.cc:482] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.391786 30965 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.394476 30965 raft_consensus.cc:504] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.395267 30965 leader_election.cc:288] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.395788 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.395795 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:01.396037 30786 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.396041 30711 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.399142 30786 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.399274 30711 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 1.
I20240503 02:13:01.399628 30828 leader_election.cc:302] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:01.399995 30965 raft_consensus.cc:2781] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.400154 30965 raft_consensus.cc:686] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:01.400463 30965 consensus_queue.cc:237] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.404006 30607 catalog_manager.cc:5484] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:01.431640 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:01.441778 30711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:01.442051 30711 raft_consensus.cc:595] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:01.443770 30711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:01.443960 30711 raft_consensus.cc:595] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS d7b4ee58dad047fe8c54f6a1ae18f0c4
I20240503 02:13:01.444343 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:13:01.444571 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:01.497912 30918 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.525967 30955 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:01.558070 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.560346 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.560665 30693 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.561342 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.571385 30788 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.571731 30786 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.571399 30861 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.575278 30913 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.576978 30967 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.578415 30970 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.579638 30860 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.579675 30861 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.579967 30788 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:01.581545 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.582409 30693 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.585465 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.588562 30972 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.588676 30967 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.589205 30972 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:01.590314 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.590687 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.618721 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.619304 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.623822 30789 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:01.624320 30692 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.624657 30684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.626200 30862 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:01.627676 30685 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.630811 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.633406 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.636657 30967 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240503 02:13:01.637676 30913 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20240503 02:13:01.647828 30684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.666419 30785 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.676060 30958 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240503 02:13:01.682111 30955 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20240503 02:13:01.690848 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.710283 30967 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.716069 30711 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.719859 30953 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
W20240503 02:13:01.724692 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.726689 30684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.731652 30916 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.735495 30965 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240503 02:13:01.744606 30918 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
I20240503 02:13:01.736510 30966 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.071s
W20240503 02:13:01.756786 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.761091 30783 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.761606 30713 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.762115 30968 raft_consensus.cc:979] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:01.762320 30968 raft_consensus.cc:1067] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:01.764282 30859 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:47064
I20240503 02:13:01.764580 30859 raft_consensus.cc:482] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:01.764779 30859 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:01.765376 31005 log.cc:924] Time spent T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240503 02:13:01.766273 30958 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:01.759646 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.769755 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.777664 30965 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20240503 02:13:01.808370 30684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.820145 30859 raft_consensus.cc:504] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:01.814723 30953 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240503 02:13:01.829572 30966 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:01.832644 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.837105 30712 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.853190 30859 leader_election.cc:288] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:01.853652 30916 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20240503 02:13:01.862151 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.878207 30692 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.898710 30685 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.910548 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
W20240503 02:13:01.913895 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.915400 30966 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
W20240503 02:13:01.915756 30692 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.915858 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.916292 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:01.928469 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.912518 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cc6188939e4b728b1c1efea836a7bb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:01.942885 30708 raft_consensus.cc:3032] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:01.958074 30708 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:01.958478 30708 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
W20240503 02:13:01.963542 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:01.984282 30966 raft_consensus.cc:979] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:02.004931 30708 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:02.005887 30687 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:02.012943 30966 raft_consensus.cc:1067] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:02.013496 31009 mvcc.cc:205] Tried to move back new op lower bound from 7023420955638530048 to 7023420954219241472. Current Snapshot: MvccSnapshot[applied={T|T < 7023420955489923072}]
I20240503 02:13:02.022204 30787 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42342
I20240503 02:13:02.022558 30787 raft_consensus.cc:482] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:02.030917 30787 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:02.035221 30685 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.057425 30692 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.060250 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.080705 30691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.086632 30689 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.096307 30684 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:02.100646 30708 raft_consensus.cc:2454] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 2.
W20240503 02:13:02.110695 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.113950 31017 log.cc:924] Time spent T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
W20240503 02:13:02.144977 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.149235 30767 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42366: Illegal state: replica 7a9cbb0303f446618b0e8bc7ad35afb4 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.150005 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
I20240503 02:13:02.151489 30829 leader_election.cc:302] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters: 7a9cbb0303f446618b0e8bc7ad35afb4
W20240503 02:13:02.151721 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.152031 30829 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.178.65:40107, user_credentials={real_user=slave}} blocked reactor thread for 46042us
I20240503 02:13:02.152572 30916 raft_consensus.cc:2781] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 02:13:02.151500 30765 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42366: Illegal state: replica 7a9cbb0303f446618b0e8bc7ad35afb4 is not leader of this config: current role FOLLOWER
I20240503 02:13:02.146739 30787 raft_consensus.cc:504] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
W20240503 02:13:02.159078 30769 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42366: Illegal state: replica 7a9cbb0303f446618b0e8bc7ad35afb4 is not leader of this config: current role FOLLOWER
I20240503 02:13:02.172688 30787 leader_election.cc:288] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:02.176311 30916 raft_consensus.cc:686] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
W20240503 02:13:02.177085 30838 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47068: Service unavailable: leader is not yet ready
I20240503 02:13:02.179458 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
W20240503 02:13:02.181061 30839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47068: Service unavailable: leader is not yet ready
W20240503 02:13:02.182132 30838 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47068: Service unavailable: leader is not yet ready
I20240503 02:13:02.176621 30916 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
W20240503 02:13:02.189034 30688 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
W20240503 02:13:02.189865 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Service unavailable: leader transfer in progress
I20240503 02:13:02.189787 30610 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.190339 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:02.190524 30709 raft_consensus.cc:3032] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:02.190596 30709 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:02.190860 30709 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:02.191100 30787 raft_consensus.cc:3037] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:02.191211 30709 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:02.193874 30709 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:02.194258 30754 leader_election.cc:302] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters: d7b4ee58dad047fe8c54f6a1ae18f0c4
I20240503 02:13:02.194581 30918 raft_consensus.cc:2781] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:02.194777 30918 raft_consensus.cc:686] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:02.195034 30918 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:02.196146 30787 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:02.196978 30916 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:02.201417 30603 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:02.202219 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.205648 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
I20240503 02:13:02.205826 30711 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:02.206295 30965 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:02.207511 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
W20240503 02:13:02.208328 30690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60528: Illegal state: replica ebd1bb9e6d2f4384b781596ce1795b54 is not leader of this config: current role FOLLOWER
I20240503 02:13:02.219744 30858 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:02.221766 30711 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:02.222302 30918 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:02.222793 30858 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:02.225055 30918 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:02.231642 30996 mvcc.cc:205] Tried to move back new op lower bound from 7023420957566849024 to 7023420957473394688. Current Snapshot: MvccSnapshot[applied={T|T < 7023420957566849024}]
W20240503 02:13:02.337746 31016 log.cc:924] Time spent T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20240503 02:13:02.467561 30961 log.cc:924] Time spent T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20240503 02:13:02.783648 30677 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.178.67:44143, user_credentials={real_user=slave}} blocked reactor thread for 52768.4us
W20240503 02:13:02.796121 30678 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.178.66:46225, user_credentials={real_user=slave}} blocked reactor thread for 68954.6us
I20240503 02:13:03.257022 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:03.261885 31061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:03.262034 31060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:03.271952 27337 server_base.cc:1034] running on GCE node
W20240503 02:13:03.273169 31063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:03.273574 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:03.273636 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:03.273670 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702383273668 us; error 0 us; skew 500 ppm
I20240503 02:13:03.273778 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:03.275164 27337 webserver.cc:460] Webserver started at http://127.26.178.68:41913/ using document root <none> and password file <none>
I20240503 02:13:03.275355 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:03.275413 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:03.275501 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:03.276074 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-3-root/instance:
uuid: "4c1727d9448f4a9c81e2c532e5339801"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.278631 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:03.280375 31068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.280853 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:13:03.280965 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-3-root
uuid: "4c1727d9448f4a9c81e2c532e5339801"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.281076 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:03.304327 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:03.304901 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:03.305437 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:03.306360 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:03.306413 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.306483 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:03.306524 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.323514 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.68:40457
I20240503 02:13:03.323565 31131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.68:40457 every 8 connection(s)
I20240503 02:13:03.325491 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:03.329389 31136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:03.330132 31137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:03.331295 31139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:03.332260 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:03.332471 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:03.332510 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:03.332541 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702383332537 us; error 0 us; skew 500 ppm
I20240503 02:13:03.332686 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:03.334239 27337 webserver.cc:460] Webserver started at http://127.26.178.69:38701/ using document root <none> and password file <none>
I20240503 02:13:03.334358 31132 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:13:03.334478 31132 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:03.334530 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:03.334627 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:03.334796 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:03.334899 31132 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:13:03.335610 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-4-root/instance:
uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.335855 30609 ts_manager.cc:194] Registered new tserver with Master: 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457)
I20240503 02:13:03.336750 30609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33278
I20240503 02:13:03.338394 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:03.340479 31144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.340914 27337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:03.341006 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-4-root
uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.341112 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:03.353348 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:03.353929 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:03.354528 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:03.355492 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:03.355547 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.355614 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:03.355666 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.373037 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.69:43601
I20240503 02:13:03.373087 31206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.69:43601 every 8 connection(s)
I20240503 02:13:03.374876 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:03.379506 31210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:03.380122 31211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:03.381247 31213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:03.381773 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:03.382041 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:03.382128 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:03.382159 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702383382155 us; error 0 us; skew 500 ppm
I20240503 02:13:03.382382 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:03.383023 31207 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:13:03.383150 31207 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:03.383656 31207 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:13:03.384207 27337 webserver.cc:460] Webserver started at http://127.26.178.70:43355/ using document root <none> and password file <none>
I20240503 02:13:03.384430 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:03.384505 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:03.384620 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:03.384619 30609 ts_manager.cc:194] Registered new tserver with Master: cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601)
I20240503 02:13:03.385298 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-5-root/instance:
uuid: "3e8a84e16aa2470899683bc9edef9283"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.385486 30609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33280
I20240503 02:13:03.388375 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:03.390120 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.390563 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:13:03.390694 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-5-root
uuid: "3e8a84e16aa2470899683bc9edef9283"
format_stamp: "Formatted at 2024-05-03 02:13:03 on dist-test-slave-0cbc"
I20240503 02:13:03.390806 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:03.420379 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:03.420943 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:03.421541 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:03.422402 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:03.422461 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.422547 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:03.422591 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:03.439424 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.70:46209
I20240503 02:13:03.439460 31280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.70:46209 every 8 connection(s)
I20240503 02:13:03.445000 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:03.447669 31281 heartbeater.cc:347] Connected to a master server at 127.26.178.126:43777
I20240503 02:13:03.447816 31281 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:03.448132 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:13:03.448298 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:03.448319 31281 heartbeater.cc:511] Master 127.26.178.126:43777 requested a full tablet report, sending...
I20240503 02:13:03.449196 30609 ts_manager.cc:194] Registered new tserver with Master: 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209)
I20240503 02:13:03.449918 30609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33292
I20240503 02:13:03.606775 30707 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:03.608907 30784 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:03.609099 30862 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:03.609717 31057 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.610023 31056 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.612926 30968 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.613654 30784 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.613863 30855 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.620277 30609 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:03.620926 30679 consensus_peers.cc:485] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c81f82e2608a401ba8f4f381c3ad95be. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.629773 30855 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:03.632035 30707 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:13:03.632062 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:13:03.632817 31046 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:13:03.633697 31046 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:13:03.635399 30958 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.637014 30784 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.639361 30707 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
W20240503 02:13:03.642153 30827 consensus_peers.cc:485] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2e4475d5b86469e90a10f84a910163a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.643035 30610 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index -1 to 27, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.652523 30784 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:03.654361 30707 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:03.655050 30855 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:03.655544 30953 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:03.656035 30955 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:03.658481 30953 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:03.661787 30707 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:03.662760 30855 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
W20240503 02:13:03.667181 30751 consensus_peers.cc:485] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ec68f05310c4dcc8e02dadf0b4934ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.667630 30784 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:03.668509 30603 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:03.670745 30862 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:03.671861 30955 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.674304 30953 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.674953 30855 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.674329 30707 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240503 02:13:03.679441 30752 consensus_peers.cc:485] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b2a8cda34b54f4482afed13d9e43c4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.679548 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index -1 to 28, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.682898 30918 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.685143 30707 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:03.687229 30855 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:03.687426 30784 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240503 02:13:03.687640 30679 consensus_peers.cc:485] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8c3012068484fe3acd108cc36eb62d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.687693 30966 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.687960 30968 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:03.689846 30970 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.690428 30855 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.690428 30784 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.696336 30609 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index -1 to 28, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:03.697135 30707 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:03.698520 30855 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:03.702426 30707 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:13:03.702816 30784 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:13:03.702883 30958 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:13:03.703263 31041 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:13:03.705469 31040 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
W20240503 02:13:03.708987 30826 consensus_peers.cc:485] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f15a775e24c46eda9ad8f5e8a537da4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.709545 30781 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:03.709561 30707 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:03.711524 30610 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index -1 to 27, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.095980 31298 ts_tablet_manager.cc:902] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:04.097252 31298 tablet_copy_client.cc:308] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.66:46225
I20240503 02:13:04.103521 30799 tablet_copy_service.cc:140] P 7a9cbb0303f446618b0e8bc7ad35afb4: Received BeginTabletCopySession request for tablet 9b2a8cda34b54f4482afed13d9e43c4f from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:42386)
I20240503 02:13:04.103669 30799 tablet_copy_service.cc:161] P 7a9cbb0303f446618b0e8bc7ad35afb4: Beginning new tablet copy session on tablet 9b2a8cda34b54f4482afed13d9e43c4f from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:42386: session id = 4c1727d9448f4a9c81e2c532e5339801-9b2a8cda34b54f4482afed13d9e43c4f
I20240503 02:13:04.106102 30799 tablet_copy_source_session.cc:205] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.107228 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a8cda34b54f4482afed13d9e43c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.112285 31298 tablet_copy_client.cc:783] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.112510 31298 tablet_copy_client.cc:647] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.113952 31302 tablet_copy_client.cc:1002] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.145241 31304 ts_tablet_manager.cc:902] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:04.146797 31304 tablet_copy_client.cc:308] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:04.153002 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet 3f15a775e24c46eda9ad8f5e8a537da4 from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:47080)
I20240503 02:13:04.153162 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet 3f15a775e24c46eda9ad8f5e8a537da4 from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:47080: session id = cf667d67d5b341c7af3b984e0c1f2d55-3f15a775e24c46eda9ad8f5e8a537da4
I20240503 02:13:04.155881 30873 tablet_copy_source_session.cc:205] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.157070 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f15a775e24c46eda9ad8f5e8a537da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.163152 31304 tablet_copy_client.cc:783] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.163388 31304 tablet_copy_client.cc:647] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.164866 31308 tablet_copy_client.cc:1002] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.217756 31311 ts_tablet_manager.cc:902] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:04.218948 31311 tablet_copy_client.cc:308] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.66:46225
I20240503 02:13:04.224058 30799 tablet_copy_service.cc:140] P 7a9cbb0303f446618b0e8bc7ad35afb4: Received BeginTabletCopySession request for tablet 4ec68f05310c4dcc8e02dadf0b4934ce from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:42398)
I20240503 02:13:04.224206 30799 tablet_copy_service.cc:161] P 7a9cbb0303f446618b0e8bc7ad35afb4: Beginning new tablet copy session on tablet 4ec68f05310c4dcc8e02dadf0b4934ce from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:42398: session id = cf667d67d5b341c7af3b984e0c1f2d55-4ec68f05310c4dcc8e02dadf0b4934ce
I20240503 02:13:04.226385 30799 tablet_copy_source_session.cc:205] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.227241 31316 ts_tablet_manager.cc:902] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:04.227596 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec68f05310c4dcc8e02dadf0b4934ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.228727 31316 tablet_copy_client.cc:308] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.65:40107
I20240503 02:13:04.234196 31311 tablet_copy_client.cc:783] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.234423 31311 tablet_copy_client.cc:647] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.236066 31315 tablet_copy_client.cc:1002] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.255288 31323 ts_tablet_manager.cc:902] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:04.257907 31323 tablet_copy_client.cc:308] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:04.258715 30723 tablet_copy_service.cc:140] P ebd1bb9e6d2f4384b781596ce1795b54: Received BeginTabletCopySession request for tablet c81f82e2608a401ba8f4f381c3ad95be from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:60554)
I20240503 02:13:04.258884 30723 tablet_copy_service.cc:161] P ebd1bb9e6d2f4384b781596ce1795b54: Beginning new tablet copy session on tablet c81f82e2608a401ba8f4f381c3ad95be from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:60554: session id = 3e8a84e16aa2470899683bc9edef9283-c81f82e2608a401ba8f4f381c3ad95be
I20240503 02:13:04.262241 30723 tablet_copy_source_session.cc:205] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.263653 31316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81f82e2608a401ba8f4f381c3ad95be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.264117 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet c2e4475d5b86469e90a10f84a910163a from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:47086)
I20240503 02:13:04.264252 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet c2e4475d5b86469e90a10f84a910163a from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:47086: session id = 4c1727d9448f4a9c81e2c532e5339801-c2e4475d5b86469e90a10f84a910163a
I20240503 02:13:04.266108 30873 tablet_copy_source_session.cc:205] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.267211 31323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e4475d5b86469e90a10f84a910163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.269969 31316 tablet_copy_client.cc:783] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.270205 31316 tablet_copy_client.cc:647] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.271632 31318 tablet_copy_client.cc:1002] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.272719 31323 tablet_copy_client.cc:783] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.272970 31323 tablet_copy_client.cc:647] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.274214 31327 tablet_copy_client.cc:1002] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.298453 31328 ts_tablet_manager.cc:902] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:04.299897 31328 tablet_copy_client.cc:308] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.65:40107
I20240503 02:13:04.300477 30723 tablet_copy_service.cc:140] P ebd1bb9e6d2f4384b781596ce1795b54: Received BeginTabletCopySession request for tablet c8c3012068484fe3acd108cc36eb62d5 from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:60554)
I20240503 02:13:04.300629 30723 tablet_copy_service.cc:161] P ebd1bb9e6d2f4384b781596ce1795b54: Beginning new tablet copy session on tablet c8c3012068484fe3acd108cc36eb62d5 from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:60554: session id = 3e8a84e16aa2470899683bc9edef9283-c8c3012068484fe3acd108cc36eb62d5
I20240503 02:13:04.302853 30723 tablet_copy_source_session.cc:205] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.303912 31328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c3012068484fe3acd108cc36eb62d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.309962 31328 tablet_copy_client.cc:783] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.310253 31328 tablet_copy_client.cc:647] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.311897 31332 tablet_copy_client.cc:1002] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:04.337852 31132 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:04.387012 31207 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:04.451043 31281 heartbeater.cc:503] Master 127.26.178.126:43777 was elected leader, sending a full tablet report...
I20240503 02:13:05.448637 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:05.451280 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:13:05.451525 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:06.114353 31298 tablet_copy_client.cc:523] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.117516 31298 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:06.160749 31298 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.161212 31298 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:06.161489 31298 ts_tablet_manager.cc:1367] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.044s user 0.031s sys 0.012s
I20240503 02:13:06.162174 31298 raft_consensus.cc:348] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:06.162374 31298 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:06.162575 31298 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:06.163542 31298 ts_tablet_manager.cc:1398] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 02:13:06.164134 30799 tablet_copy_service.cc:342] P 7a9cbb0303f446618b0e8bc7ad35afb4: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-9b2a8cda34b54f4482afed13d9e43c4f received from {username='slave'} at 127.0.0.1:42386
I20240503 02:13:06.164245 30799 tablet_copy_service.cc:433] P 7a9cbb0303f446618b0e8bc7ad35afb4: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-9b2a8cda34b54f4482afed13d9e43c4f on tablet 9b2a8cda34b54f4482afed13d9e43c4f with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:06.165323 31304 tablet_copy_client.cc:523] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.168152 31304 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:06.179890 31107 raft_consensus.cc:1203] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 02:13:06.203203 31304 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.203644 31304 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:06.203881 31304 ts_tablet_manager.cc:1367] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.036s user 0.031s sys 0.003s
I20240503 02:13:06.204533 31304 raft_consensus.cc:348] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:06.204720 31304 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:06.204895 31304 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:06.205624 31304 ts_tablet_manager.cc:1398] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:06.206187 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-3f15a775e24c46eda9ad8f5e8a537da4 received from {username='slave'} at 127.0.0.1:47080
I20240503 02:13:06.206307 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-3f15a775e24c46eda9ad8f5e8a537da4 on tablet 3f15a775e24c46eda9ad8f5e8a537da4 with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:06.236574 31311 tablet_copy_client.cc:523] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.240099 31311 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:06.272097 31316 tablet_copy_client.cc:523] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.274647 31323 tablet_copy_client.cc:523] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.277042 31316 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:06.277432 31323 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:06.293777 31311 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.294255 31311 tablet_bootstrap.cc:492] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:06.294574 31311 ts_tablet_manager.cc:1367] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.055s user 0.050s sys 0.004s
I20240503 02:13:06.295692 31311 raft_consensus.cc:348] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:06.295924 31311 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:06.296093 31311 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:06.297314 31311 ts_tablet_manager.cc:1398] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:06.298861 30799 tablet_copy_service.cc:342] P 7a9cbb0303f446618b0e8bc7ad35afb4: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-4ec68f05310c4dcc8e02dadf0b4934ce received from {username='slave'} at 127.0.0.1:42398
I20240503 02:13:06.298964 30799 tablet_copy_service.cc:433] P 7a9cbb0303f446618b0e8bc7ad35afb4: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-4ec68f05310c4dcc8e02dadf0b4934ce on tablet 4ec68f05310c4dcc8e02dadf0b4934ce with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:06.312265 31328 tablet_copy_client.cc:523] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.315907 31328 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:06.325352 31323 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.325760 31323 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:06.326021 31323 ts_tablet_manager.cc:1367] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.049s user 0.044s sys 0.000s
I20240503 02:13:06.326773 31323 raft_consensus.cc:348] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:06.327010 31323 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:06.327200 31323 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:06.327680 31316 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.327714 31323 ts_tablet_manager.cc:1398] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:06.327994 31316 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:06.328231 31316 ts_tablet_manager.cc:1367] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.051s user 0.030s sys 0.016s
I20240503 02:13:06.328963 31316 raft_consensus.cc:348] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:06.329180 31316 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:06.329473 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-c2e4475d5b86469e90a10f84a910163a received from {username='slave'} at 127.0.0.1:47086
I20240503 02:13:06.329576 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-c2e4475d5b86469e90a10f84a910163a on tablet c2e4475d5b86469e90a10f84a910163a with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:06.329427 31316 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:06.334343 31316 ts_tablet_manager.cc:1398] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:13:06.335317 30723 tablet_copy_service.cc:342] P ebd1bb9e6d2f4384b781596ce1795b54: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-c81f82e2608a401ba8f4f381c3ad95be received from {username='slave'} at 127.0.0.1:60554
I20240503 02:13:06.335409 30723 tablet_copy_service.cc:433] P ebd1bb9e6d2f4384b781596ce1795b54: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-c81f82e2608a401ba8f4f381c3ad95be on tablet c81f82e2608a401ba8f4f381c3ad95be with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:06.365056 31328 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.365633 31328 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:06.365903 31328 ts_tablet_manager.cc:1367] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.050s user 0.034s sys 0.006s
I20240503 02:13:06.366489 31328 raft_consensus.cc:348] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:06.366752 31328 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:06.366995 31328 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:06.367550 31328 ts_tablet_manager.cc:1398] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:13:06.368177 30723 tablet_copy_service.cc:342] P ebd1bb9e6d2f4384b781596ce1795b54: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-c8c3012068484fe3acd108cc36eb62d5 received from {username='slave'} at 127.0.0.1:60554
I20240503 02:13:06.368280 30723 tablet_copy_service.cc:433] P ebd1bb9e6d2f4384b781596ce1795b54: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-c8c3012068484fe3acd108cc36eb62d5 on tablet c8c3012068484fe3acd108cc36eb62d5 with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:06.509712 31256 raft_consensus.cc:1203] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 02:13:06.603798 31107 raft_consensus.cc:1203] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:13:06.703720 31182 raft_consensus.cc:1203] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:13:06.711359 31349 raft_consensus.cc:1050] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:06.712066 31349 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:06.714277 31107 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:06.714282 30711 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:06.714490 30855 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:06.714905 31348 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:06.715404 31333 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:06.716012 31333 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:06.720093 31333 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.723181 31107 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.727741 30609 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 28 to 29, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.736848 30862 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.737105 30711 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.743852 30784 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:06.745404 31107 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:06.745889 31351 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:06.746066 30862 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:06.747010 31351 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:06.751392 31256 raft_consensus.cc:1203] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 02:13:06.749637 31351 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.753132 31107 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.756482 30596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b2a8cda34b54f4482afed13d9e43c4f with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:06.758381 30609 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 29 to 30, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:06.757912 30862 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:06.765061 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2a8cda34b54f4482afed13d9e43c4f with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:06.765800 31359 tablet_replica.cc:330] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:06.766005 31359 raft_consensus.cc:2229] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:06.766260 31359 raft_consensus.cc:2258] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.767236 31359 ts_tablet_manager.cc:1875] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:06.772751 31359 ts_tablet_manager.cc:1888] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 02:13:06.772881 31359 log.cc:1196] T 9b2a8cda34b54f4482afed13d9e43c4f P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/9b2a8cda34b54f4482afed13d9e43c4f
I20240503 02:13:06.773577 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet 9b2a8cda34b54f4482afed13d9e43c4f (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:06.827255 31182 raft_consensus.cc:1203] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 02:13:07.024609 31361 raft_consensus.cc:1050] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:07.025544 31361 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.028105 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.028107 30784 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.028765 31363 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.029129 30855 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.029157 31361 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.029671 31360 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.033319 31363 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.033974 31256 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.035094 30862 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.039391 30603 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.040225 30784 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.047631 30711 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.049748 30784 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:07.050526 31363 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.050948 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:07.051558 31363 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.053462 31337 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.055315 30784 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.055347 31256 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.056972 30597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8c3012068484fe3acd108cc36eb62d5 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.058310 30609 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 29 to 30, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.061656 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c3012068484fe3acd108cc36eb62d5 with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:07.062359 31367 tablet_replica.cc:330] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:07.062597 31367 raft_consensus.cc:2229] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.063139 31367 raft_consensus.cc:2258] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.064132 31367 ts_tablet_manager.cc:1875] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.070408 31367 ts_tablet_manager.cc:1888] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 02:13:07.070541 31367 log.cc:1196] T c8c3012068484fe3acd108cc36eb62d5 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/c8c3012068484fe3acd108cc36eb62d5
I20240503 02:13:07.071218 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet c8c3012068484fe3acd108cc36eb62d5 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:07.173025 30958 raft_consensus.cc:1050] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:07.173807 30958 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.179879 31182 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.180012 30784 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.180491 30707 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.180603 30958 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.181010 31368 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.181401 31371 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.184170 31368 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.184671 30784 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.190037 30605 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 27 to 28, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.191648 31182 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.191712 30707 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.196456 30862 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.198961 31371 raft_consensus.cc:1050] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:07.199807 31371 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:07.200448 31182 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.201124 31368 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.203518 30711 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.204407 31107 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.204843 31368 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.205103 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:07.205307 31375 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.205565 30958 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.207113 30712 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.208242 31182 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.208845 30958 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.209514 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f15a775e24c46eda9ad8f5e8a537da4 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.210574 30712 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.211629 30605 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 28 to 29, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.216243 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f15a775e24c46eda9ad8f5e8a537da4 with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:07.216847 31379 tablet_replica.cc:330] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:07.217168 31379 raft_consensus.cc:2229] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.218919 31379 raft_consensus.cc:2258] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.221496 31375 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:07.224022 31379 ts_tablet_manager.cc:1875] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.238754 31379 ts_tablet_manager.cc:1888] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:13:07.238891 31379 log.cc:1196] T 3f15a775e24c46eda9ad8f5e8a537da4 P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/3f15a775e24c46eda9ad8f5e8a537da4
I20240503 02:13:07.239100 30958 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.239698 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet 3f15a775e24c46eda9ad8f5e8a537da4 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:07.240366 30712 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.241186 30787 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.245226 31107 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.245460 30610 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 27 to 28, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:07.250386 30958 consensus_queue.cc:1159] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4c1727d9448f4a9c81e2c532e5339801" responder_term: 1 status { last_received { term: 1 index: 28 } last_committed_idx: 28 last_received_current_leader { term: 1 index: 28 } }
I20240503 02:13:07.251259 30862 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:07.253065 31107 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.253463 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.253564 30958 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.253939 31371 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.255591 30958 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.256404 31107 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.257508 30784 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.260311 30610 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 28 to 29, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:07.262699 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2e4475d5b86469e90a10f84a910163a with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.264556 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e4475d5b86469e90a10f84a910163a with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:07.264843 31359 tablet_replica.cc:330] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:07.265009 31359 raft_consensus.cc:2229] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.265262 31359 raft_consensus.cc:2258] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.266076 31359 ts_tablet_manager.cc:1875] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.271363 31359 ts_tablet_manager.cc:1888] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:13:07.271481 31359 log.cc:1196] T c2e4475d5b86469e90a10f84a910163a P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/c2e4475d5b86469e90a10f84a910163a
I20240503 02:13:07.272452 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet c2e4475d5b86469e90a10f84a910163a (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:07.303632 31382 raft_consensus.cc:1050] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:07.304376 31382 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.306686 31182 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:07.307107 30855 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:07.307229 31333 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.307139 30712 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:07.307683 31381 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20240503 02:13:07.308853 31333 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.320005 31333 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.320828 30855 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.320818 30712 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.324738 30609 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 29 to 30, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.323613 31182 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.330596 30784 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.332319 31182 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:07.332425 30712 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:07.332798 31333 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:07.333084 31382 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:07.335281 31381 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.335951 31182 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.336098 30712 raft_consensus.cc:2932] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:07.339357 30596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ec68f05310c4dcc8e02dadf0b4934ce with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.340991 30610 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.344959 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec68f05310c4dcc8e02dadf0b4934ce with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:07.345288 31367 tablet_replica.cc:330] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:07.345494 31367 raft_consensus.cc:2229] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.345851 31367 raft_consensus.cc:2258] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.346635 31367 ts_tablet_manager.cc:1875] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.348410 31363 raft_consensus.cc:1050] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:07.349186 31363 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.351575 30784 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.351583 31256 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.351832 30855 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:13:07.352252 31360 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.352492 31361 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.352670 31337 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:07.353258 31367 ts_tablet_manager.cc:1888] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 02:13:07.353390 31367 log.cc:1196] T 4ec68f05310c4dcc8e02dadf0b4934ce P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/4ec68f05310c4dcc8e02dadf0b4934ce
I20240503 02:13:07.354143 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet 4ec68f05310c4dcc8e02dadf0b4934ce (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:07.360199 31360 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.361268 30855 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.364935 31256 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.366789 30609 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.368568 30784 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.371858 30712 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.373427 30855 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:07.373883 31256 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:07.374218 31360 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.374807 31361 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:07.377607 31363 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.379029 30855 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.379427 31256 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:07.381143 30597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c81f82e2608a401ba8f4f381c3ad95be with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.382591 30610 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 29 to 30, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.386824 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet c81f82e2608a401ba8f4f381c3ad95be with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:07.387233 31379 tablet_replica.cc:330] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:07.387396 31379 raft_consensus.cc:2229] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.387639 31379 raft_consensus.cc:2258] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.388382 31379 ts_tablet_manager.cc:1875] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.392719 31379 ts_tablet_manager.cc:1888] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 02:13:07.392815 31379 log.cc:1196] T c81f82e2608a401ba8f4f381c3ad95be P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/c81f82e2608a401ba8f4f381c3ad95be
I20240503 02:13:07.393406 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet c81f82e2608a401ba8f4f381c3ad95be (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:07.451958 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:07.467098 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
new_leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:42406
I20240503 02:13:07.467368 30784 raft_consensus.cc:595] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Received request to transfer leadership to TS ebd1bb9e6d2f4384b781596ce1795b54
I20240503 02:13:07.468467 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
new_leader_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
from {username='slave'} at 127.0.0.1:42406
I20240503 02:13:07.468614 30784 raft_consensus.cc:595] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Received request to transfer leadership to TS cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:07.474889 30855 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:47090
I20240503 02:13:07.475073 30855 raft_consensus.cc:595] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:07.476176 30855 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
new_leader_uuid: "4c1727d9448f4a9c81e2c532e5339801"
from {username='slave'} at 127.0.0.1:47090
I20240503 02:13:07.476368 30855 raft_consensus.cc:595] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Received request to transfer leadership to TS 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:07.477304 30712 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "3e8a84e16aa2470899683bc9edef9283"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:07.477491 30712 raft_consensus.cc:595] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:07.478408 30712 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:07.478542 30712 raft_consensus.cc:595] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:07.478930 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 02:13:07.479140 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:07.626001 31371 raft_consensus.cc:979] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:07.626152 31371 raft_consensus.cc:1067] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:07.626863 30784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42354
I20240503 02:13:07.627050 30784 raft_consensus.cc:482] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.627123 30784 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.629115 30784 raft_consensus.cc:504] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.629681 30784 leader_election.cc:288] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:07.630113 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:07.630081 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5277bc5b18a0494baf73181fb4eeba01" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:07.630316 30712 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.630335 30855 raft_consensus.cc:3032] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:07.630475 30855 raft_consensus.cc:729] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:07.630760 30855 consensus_queue.cc:260] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.631157 30855 raft_consensus.cc:3037] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.633457 30712 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:07.633910 30754 leader_election.cc:302] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:07.633996 30855 raft_consensus.cc:2454] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:07.634203 31333 raft_consensus.cc:2781] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:07.634361 31333 raft_consensus.cc:686] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:07.634619 31333 consensus_queue.cc:237] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.637210 30605 catalog_manager.cc:5484] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 1 to 2, leader changed from d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.666882 31363 raft_consensus.cc:979] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:07.667027 31363 raft_consensus.cc:1067] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:07.667626 30784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42342
I20240503 02:13:07.667812 30784 raft_consensus.cc:482] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.667905 30784 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.670017 30784 raft_consensus.cc:504] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.670631 30784 leader_election.cc:288] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209)
I20240503 02:13:07.671078 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:07.671234 30712 raft_consensus.cc:3032] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:07.671329 30712 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:07.671590 30712 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.671850 30712 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.673987 30712 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:07.674945 30754 leader_election.cc:302] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:07.675264 31333 raft_consensus.cc:2781] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:07.675407 31333 raft_consensus.cc:686] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:07.675637 31333 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.678628 30605 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.682739 31256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c3012068484fe3acd108cc36eb62d5" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "3e8a84e16aa2470899683bc9edef9283"
I20240503 02:13:07.682904 31256 raft_consensus.cc:3037] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.684885 31256 raft_consensus.cc:2454] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:07.695962 30958 raft_consensus.cc:979] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: : Instructing follower 4c1727d9448f4a9c81e2c532e5339801 to start an election
I20240503 02:13:07.696108 30958 raft_consensus.cc:1067] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Signalling peer 4c1727d9448f4a9c81e2c532e5339801 to start an election
I20240503 02:13:07.696729 31107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a"
dest_uuid: "4c1727d9448f4a9c81e2c532e5339801"
from {username='slave'} at 127.0.0.1:39878
I20240503 02:13:07.696884 31107 raft_consensus.cc:482] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.696972 31107 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.699086 31107 raft_consensus.cc:504] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:07.699985 31107 leader_election.cc:288] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [CANDIDATE]: Term 2 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:07.700429 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "4c1727d9448f4a9c81e2c532e5339801" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:07.700430 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e4475d5b86469e90a10f84a910163a" candidate_uuid: "4c1727d9448f4a9c81e2c532e5339801" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:07.700639 30855 raft_consensus.cc:3032] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:07.700714 30855 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:07.700639 30784 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.700984 30855 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:07.701337 30855 raft_consensus.cc:3037] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.703940 30784 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1727d9448f4a9c81e2c532e5339801 in term 2.
I20240503 02:13:07.704134 30855 raft_consensus.cc:2454] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1727d9448f4a9c81e2c532e5339801 in term 2.
I20240503 02:13:07.704396 31071 leader_election.cc:302] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1727d9448f4a9c81e2c532e5339801, 7a9cbb0303f446618b0e8bc7ad35afb4; no voters:
I20240503 02:13:07.704655 31340 raft_consensus.cc:2781] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:07.705040 31340 raft_consensus.cc:686] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEADER]: Becoming Leader. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Running, Role: LEADER
I20240503 02:13:07.705348 31340 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:07.707510 30605 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: term changed from 1 to 2, leader changed from d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) to 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68). New cstate: current_term: 2 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.791891 31361 raft_consensus.cc:979] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 3e8a84e16aa2470899683bc9edef9283 to start an election
I20240503 02:13:07.792049 31361 raft_consensus.cc:1067] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer 3e8a84e16aa2470899683bc9edef9283 to start an election
I20240503 02:13:07.792630 31256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be"
dest_uuid: "3e8a84e16aa2470899683bc9edef9283"
from {username='slave'} at 127.0.0.1:36802
I20240503 02:13:07.792795 31256 raft_consensus.cc:482] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.792876 31256 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.794827 31256 raft_consensus.cc:504] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.795522 31256 leader_election.cc:288] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:07.795954 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "3e8a84e16aa2470899683bc9edef9283" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:07.796145 30712 raft_consensus.cc:3032] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:07.796223 30712 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:07.796502 30712 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.796852 30712 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.799396 30712 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8a84e16aa2470899683bc9edef9283 in term 2.
I20240503 02:13:07.799784 31222 leader_election.cc:302] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a84e16aa2470899683bc9edef9283, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:07.800050 31346 raft_consensus.cc:2781] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:07.800489 31346 raft_consensus.cc:686] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEADER]: Becoming Leader. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Running, Role: LEADER
I20240503 02:13:07.800757 31346 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:07.802520 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81f82e2608a401ba8f4f381c3ad95be" candidate_uuid: "3e8a84e16aa2470899683bc9edef9283" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:07.802728 30855 raft_consensus.cc:3037] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.803012 30605 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70). New cstate: current_term: 2 leader_uuid: "3e8a84e16aa2470899683bc9edef9283" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.805164 30855 raft_consensus.cc:2454] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8a84e16aa2470899683bc9edef9283 in term 2.
I20240503 02:13:07.835422 31381 raft_consensus.cc:979] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower cf667d67d5b341c7af3b984e0c1f2d55 to start an election
I20240503 02:13:07.835619 31381 raft_consensus.cc:1067] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Signalling peer cf667d67d5b341c7af3b984e0c1f2d55 to start an election
I20240503 02:13:07.836290 31182 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce"
dest_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
from {username='slave'} at 127.0.0.1:57050
I20240503 02:13:07.836472 31182 raft_consensus.cc:482] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.836577 31182 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:07.838526 31182 raft_consensus.cc:504] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.839290 31182 leader_election.cc:288] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [CANDIDATE]: Term 3 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:07.839864 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:07.840093 30784 raft_consensus.cc:3032] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:07.840157 30784 raft_consensus.cc:729] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:07.840438 30784 consensus_queue.cc:260] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.840816 30784 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:07.843735 30784 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf667d67d5b341c7af3b984e0c1f2d55 in term 3.
I20240503 02:13:07.844115 31147 leader_election.cc:302] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, cf667d67d5b341c7af3b984e0c1f2d55; no voters:
I20240503 02:13:07.844799 31403 raft_consensus.cc:2781] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:07.845275 31403 raft_consensus.cc:686] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEADER]: Becoming Leader. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Running, Role: LEADER
I20240503 02:13:07.845652 31403 consensus_queue.cc:237] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:07.847659 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec68f05310c4dcc8e02dadf0b4934ce" candidate_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:07.847846 30712 raft_consensus.cc:3037] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:07.848260 30605 catalog_manager.cc:5484] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 reported cstate change: term changed from 2 to 3, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69). New cstate: current_term: 3 leader_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.850241 30712 raft_consensus.cc:2454] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf667d67d5b341c7af3b984e0c1f2d55 in term 3.
I20240503 02:13:07.963945 31381 raft_consensus.cc:979] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:07.964099 31381 raft_consensus.cc:1067] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Signalling peer ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:07.964787 30712 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:60496
I20240503 02:13:07.964954 30712 raft_consensus.cc:482] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:07.965044 30712 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.967056 30712 raft_consensus.cc:504] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.967590 30712 leader_election.cc:288] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 2 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:07.967993 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:07.967993 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441b8fdd54574e0f837d04e0916d1bf1" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:07.968200 30784 raft_consensus.cc:3032] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:07.968212 30855 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.968286 30784 raft_consensus.cc:729] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:07.968524 30784 consensus_queue.cc:260] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.968910 30784 raft_consensus.cc:3037] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:07.971256 30855 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 2.
I20240503 02:13:07.971611 30784 raft_consensus.cc:2454] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 2.
I20240503 02:13:07.971679 30677 leader_election.cc:302] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:07.971987 31361 raft_consensus.cc:2781] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:07.972095 31361 raft_consensus.cc:686] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:07.972316 31361 consensus_queue.cc:237] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:07.974737 30605 catalog_manager.cc:5484] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 1 to 2, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.147351 30855 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:08.147912 31381 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:08.151904 30712 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:08.152531 31333 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:08.185297 30862 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.185907 31346 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.189855 30712 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.190111 30855 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:08.190603 31346 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.190537 31340 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:08.193776 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:08.194221 31340 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:08.194520 30712 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.195526 31333 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.206192 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.206813 31333 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.309448 30712 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Refusing update from remote peer cf667d67d5b341c7af3b984e0c1f2d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:13:08.310096 31403 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:08.313459 30784 raft_consensus.cc:1261] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer cf667d67d5b341c7af3b984e0c1f2d55: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:13:08.313966 31404 consensus_queue.cc:1035] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:08.391139 30855 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:08.393898 31182 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:08.394429 30712 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:13:08.394913 31419 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:08.395241 31418 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:08.399510 31418 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.400190 31182 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
W20240503 02:13:08.400959 30829 consensus_peers.cc:485] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f15a775e24c46eda9ad8f5e8a537da4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.402315 30712 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.405565 30610 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 reported cstate change: config changed from index 29 to 30, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.406477 30784 consensus_queue.cc:237] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
W20240503 02:13:08.409057 30752 consensus_peers.cc:485] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5277bc5b18a0494baf73181fb4eeba01. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.409420 30855 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:08.409919 31413 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:08.409672 30712 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:08.412101 31413 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:08.412582 31413 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.413121 30855 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.415525 30712 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.418480 30605 catalog_manager.cc:5484] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.420176 30707 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:08.422786 30855 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:08.423187 30784 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:13:08.423890 31361 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:08.424232 31363 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:08.439049 31361 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.442026 30862 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.442163 30784 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.445842 30855 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:08.447793 30605 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index -1 to 28, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 1 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:08.454154 30676 consensus_peers.cc:485] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e3d6429d7dd499fbe2957fb46fdf30b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.454854 30784 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:08.455570 31361 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 02:13:08.457674 30827 consensus_peers.cc:485] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cc6188939e4b728b1c1efea836a7bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.457692 30787 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:08.457942 30707 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:08.458407 31371 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:08.458966 31419 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:08.468729 30855 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:08.469769 31418 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.471063 31425 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:08.470377 30787 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.475375 30707 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:08.475744 30784 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:08.475899 30603 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.480482 31107 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240503 02:13:08.480815 30754 consensus_peers.cc:485] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b2a8cda34b54f4482afed13d9e43c4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.481316 30862 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:13:08.481990 31413 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.482497 31413 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.485966 31413 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.486778 30862 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.491336 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 30 to 31, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 1 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.504000 31106 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:08.505918 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.506356 30862 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:08.506364 31107 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:08.507161 31398 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.507714 31398 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:08.512329 31340 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
W20240503 02:13:08.513700 31069 consensus_peers.cc:485] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2e4475d5b86469e90a10f84a910163a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.513428 30784 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.513447 30862 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.517509 30610 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 29 to 31, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 2 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:08.835909 31447 ts_tablet_manager.cc:902] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:08.837512 31447 tablet_copy_client.cc:308] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:08.838133 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet 60cc6188939e4b728b1c1efea836a7bb from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:47086)
I20240503 02:13:08.838243 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet 60cc6188939e4b728b1c1efea836a7bb from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:47086: session id = 4c1727d9448f4a9c81e2c532e5339801-60cc6188939e4b728b1c1efea836a7bb
I20240503 02:13:08.840740 30873 tablet_copy_source_session.cc:205] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:08.841899 31447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cc6188939e4b728b1c1efea836a7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.847579 31447 tablet_copy_client.cc:783] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.847800 31447 tablet_copy_client.cc:647] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:08.849349 31451 tablet_copy_client.cc:1002] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:08.884280 31452 ts_tablet_manager.cc:902] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:08.885851 31452 tablet_copy_client.cc:308] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.66:46225
I20240503 02:13:08.899080 30799 tablet_copy_service.cc:140] P 7a9cbb0303f446618b0e8bc7ad35afb4: Received BeginTabletCopySession request for tablet 9b2a8cda34b54f4482afed13d9e43c4f from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:38744)
I20240503 02:13:08.899287 30799 tablet_copy_service.cc:161] P 7a9cbb0303f446618b0e8bc7ad35afb4: Beginning new tablet copy session on tablet 9b2a8cda34b54f4482afed13d9e43c4f from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:38744: session id = 3e8a84e16aa2470899683bc9edef9283-9b2a8cda34b54f4482afed13d9e43c4f
I20240503 02:13:08.902015 30799 tablet_copy_source_session.cc:205] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:08.903154 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a8cda34b54f4482afed13d9e43c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.908795 31452 tablet_copy_client.cc:783] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.909118 31452 tablet_copy_client.cc:647] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:08.910531 31456 tablet_copy_client.cc:1002] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:08.944491 31459 ts_tablet_manager.cc:902] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457)
I20240503 02:13:08.946097 31459 tablet_copy_client.cc:308] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.68:40457
I20240503 02:13:08.952656 31117 tablet_copy_service.cc:140] P 4c1727d9448f4a9c81e2c532e5339801: Received BeginTabletCopySession request for tablet c2e4475d5b86469e90a10f84a910163a from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:38980)
I20240503 02:13:08.952843 31117 tablet_copy_service.cc:161] P 4c1727d9448f4a9c81e2c532e5339801: Beginning new tablet copy session on tablet c2e4475d5b86469e90a10f84a910163a from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:38980: session id = cf667d67d5b341c7af3b984e0c1f2d55-c2e4475d5b86469e90a10f84a910163a
I20240503 02:13:08.955557 31117 tablet_copy_source_session.cc:205] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:08.956702 31459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e4475d5b86469e90a10f84a910163a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.962779 31459 tablet_copy_client.cc:783] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.963068 31459 tablet_copy_client.cc:647] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:08.964736 31463 tablet_copy_client.cc:1002] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:08.970129 31465 ts_tablet_manager.cc:902] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:08.971609 31465 tablet_copy_client.cc:308] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.66:46225
I20240503 02:13:08.972148 30799 tablet_copy_service.cc:140] P 7a9cbb0303f446618b0e8bc7ad35afb4: Received BeginTabletCopySession request for tablet 5277bc5b18a0494baf73181fb4eeba01 from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:42386)
I20240503 02:13:08.972287 30799 tablet_copy_service.cc:161] P 7a9cbb0303f446618b0e8bc7ad35afb4: Beginning new tablet copy session on tablet 5277bc5b18a0494baf73181fb4eeba01 from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:42386: session id = 4c1727d9448f4a9c81e2c532e5339801-5277bc5b18a0494baf73181fb4eeba01
I20240503 02:13:08.975183 30799 tablet_copy_source_session.cc:205] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:08.976295 31465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5277bc5b18a0494baf73181fb4eeba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.981359 31465 tablet_copy_client.cc:783] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.981657 31465 tablet_copy_client.cc:647] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:08.982935 31469 tablet_copy_client.cc:1002] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:09.005818 31470 ts_tablet_manager.cc:902] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:09.007298 31470 tablet_copy_client.cc:308] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.65:40107
I20240503 02:13:09.007899 30723 tablet_copy_service.cc:140] P ebd1bb9e6d2f4384b781596ce1795b54: Received BeginTabletCopySession request for tablet 4e3d6429d7dd499fbe2957fb46fdf30b from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:48762)
I20240503 02:13:09.008002 30723 tablet_copy_service.cc:161] P ebd1bb9e6d2f4384b781596ce1795b54: Beginning new tablet copy session on tablet 4e3d6429d7dd499fbe2957fb46fdf30b from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:48762: session id = cf667d67d5b341c7af3b984e0c1f2d55-4e3d6429d7dd499fbe2957fb46fdf30b
I20240503 02:13:09.010134 30723 tablet_copy_source_session.cc:205] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:09.011186 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d6429d7dd499fbe2957fb46fdf30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.014902 31475 ts_tablet_manager.cc:902] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:09.017510 31470 tablet_copy_client.cc:783] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:09.017773 31470 tablet_copy_client.cc:647] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:09.019469 31474 tablet_copy_client.cc:1002] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:09.023162 31475 tablet_copy_client.cc:308] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:09.023741 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet 3f15a775e24c46eda9ad8f5e8a537da4 from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:39070)
I20240503 02:13:09.023857 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet 3f15a775e24c46eda9ad8f5e8a537da4 from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:39070: session id = 3e8a84e16aa2470899683bc9edef9283-3f15a775e24c46eda9ad8f5e8a537da4
I20240503 02:13:09.026129 30873 tablet_copy_source_session.cc:205] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:09.027346 31475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f15a775e24c46eda9ad8f5e8a537da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.032714 31475 tablet_copy_client.cc:783] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:09.032979 31475 tablet_copy_client.cc:647] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:09.034531 31479 tablet_copy_client.cc:1002] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:09.479506 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:09.482939 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
new_leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:42406
I20240503 02:13:09.483319 30784 raft_consensus.cc:595] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Received request to transfer leadership to TS ebd1bb9e6d2f4384b781596ce1795b54
I20240503 02:13:09.484385 30712 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:09.484552 30712 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:09.485805 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
new_leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:42406
I20240503 02:13:09.485997 30784 raft_consensus.cc:595] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Received request to transfer leadership to TS d7b4ee58dad047fe8c54f6a1ae18f0c4
I20240503 02:13:09.486944 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 02:13:09.487213 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:09.556299 31381 raft_consensus.cc:979] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:09.556545 31381 raft_consensus.cc:1067] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Signalling peer d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:09.557174 30855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:47060
I20240503 02:13:09.557394 30855 raft_consensus.cc:482] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:09.557528 30855 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.559892 30855 raft_consensus.cc:504] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:09.560604 30855 leader_election.cc:288] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457)
I20240503 02:13:09.560951 31106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "4c1727d9448f4a9c81e2c532e5339801"
I20240503 02:13:09.561165 31106 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.561122 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:09.561513 30784 raft_consensus.cc:3032] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:09.561620 30784 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:09.561924 30784 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:09.562366 30784 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.564071 31106 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 2.
I20240503 02:13:09.564473 30827 leader_election.cc:302] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1727d9448f4a9c81e2c532e5339801, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:09.564803 31482 raft_consensus.cc:2781] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:09.564977 31482 raft_consensus.cc:686] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:09.565253 31482 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:09.567201 30784 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 2.
I20240503 02:13:09.568390 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 1 to 2, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.576359 30862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:09.576596 30862 raft_consensus.cc:595] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:13:09.583706 30862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:09.583909 30862 raft_consensus.cc:595] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:13:09.584314 30595 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 9b2a8cda34b54f4482afed13d9e43c4f already in progress [suppressed 6 similar messages]
I20240503 02:13:09.585391 30862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:09.585570 30862 raft_consensus.cc:595] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:13:09.586019 30659 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 9b2a8cda34b54f4482afed13d9e43c4f already in progress
W20240503 02:13:09.586095 30659 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 9b2a8cda34b54f4482afed13d9e43c4f already in progress
I20240503 02:13:09.642277 31413 raft_consensus.cc:979] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:09.642447 31413 raft_consensus.cc:1067] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Signalling peer ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:09.643019 30712 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:60496
I20240503 02:13:09.643208 30712 raft_consensus.cc:482] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:09.643306 30712 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.645804 30712 raft_consensus.cc:504] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:09.646415 30712 leader_election.cc:288] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 2 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:09.646739 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:09.646929 30787 raft_consensus.cc:3032] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:09.647032 30787 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:09.647298 30787 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:09.647610 30787 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.649838 30787 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 2.
I20240503 02:13:09.650120 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:09.650210 30678 leader_election.cc:302] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:09.650380 30862 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.650530 31484 raft_consensus.cc:2781] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:09.650722 31484 raft_consensus.cc:686] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:09.650944 31484 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:09.652665 30862 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 2.
I20240503 02:13:09.654220 30610 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 1 to 2, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.830523 31483 raft_consensus.cc:979] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:09.830729 31483 raft_consensus.cc:1067] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:09.831230 30787 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42342
I20240503 02:13:09.831410 30787 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:09.831506 30787 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.834028 30787 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:09.834801 30787 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Requested vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:09.835139 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:09.835320 30855 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.835264 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:09.835443 30712 raft_consensus.cc:3032] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:09.835510 30712 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:09.835731 30712 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:09.836136 30712 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:09.837517 30855 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:09.837925 30752 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:09.838021 30712 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:09.838308 31413 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:09.838459 31413 raft_consensus.cc:686] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:09.838745 31413 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:09.842792 30610 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.850950 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:09.851140 30787 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:13:09.856899 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:09.857086 30787 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:13:09.857935 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:09.858070 30787 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:13:09.858482 30659 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 4e3d6429d7dd499fbe2957fb46fdf30b already in progress
W20240503 02:13:09.858546 30659 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 4e3d6429d7dd499fbe2957fb46fdf30b already in progress
I20240503 02:13:10.002887 31106 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:10.003450 31480 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:10.006304 31482 raft_consensus.cc:979] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: : Instructing follower 4c1727d9448f4a9c81e2c532e5339801 to start an election
I20240503 02:13:10.006480 31482 raft_consensus.cc:1067] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Signalling peer 4c1727d9448f4a9c81e2c532e5339801 to start an election
I20240503 02:13:10.007205 31107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f"
dest_uuid: "4c1727d9448f4a9c81e2c532e5339801"
from {username='slave'} at 127.0.0.1:39878
I20240503 02:13:10.008143 31107 raft_consensus.cc:482] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:10.008248 31107 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:13:10.009084 30829 consensus_peers.cc:485] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:10.009086 30787 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:10.009568 31480 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:10.010936 31107 raft_consensus.cc:504] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:10.011669 31107 leader_election.cc:288] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [CANDIDATE]: Term 3 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:10.011857 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "4c1727d9448f4a9c81e2c532e5339801" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:10.012239 30862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a8cda34b54f4482afed13d9e43c4f" candidate_uuid: "4c1727d9448f4a9c81e2c532e5339801" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:10.012404 30862 raft_consensus.cc:3032] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:10.012493 30862 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:10.012769 30862 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:10.013103 30862 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.015065 30862 raft_consensus.cc:2454] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1727d9448f4a9c81e2c532e5339801 in term 3.
I20240503 02:13:10.015416 31070 leader_election.cc:302] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c1727d9448f4a9c81e2c532e5339801, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters: 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:10.015697 31485 raft_consensus.cc:2781] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:10.029047 31485 raft_consensus.cc:686] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 LEADER]: Becoming Leader. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Running, Role: LEADER
I20240503 02:13:10.029466 31485 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:10.031925 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: term changed from 2 to 3, leader changed from d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) to 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68). New cstate: current_term: 3 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.059408 30862 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:10.060074 31363 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:10.064356 30787 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:10.064833 31483 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 02:13:10.252902 30751 consensus_peers.cc:485] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:10.300110 30712 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:10.300686 31502 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:10.303694 31381 raft_consensus.cc:979] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:10.303850 31381 raft_consensus.cc:1067] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Signalling peer ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:10.305030 30707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:60496
I20240503 02:13:10.305224 30707 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:10.305327 30707 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.305735 30862 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:10.306926 31381 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:10.308038 30707 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:10.308574 30707 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 3 election: Requested vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:10.309031 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:10.309201 30784 raft_consensus.cc:3032] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:10.309257 30784 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:10.309548 30784 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:10.309950 30784 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.312650 30784 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 3.
I20240503 02:13:10.312929 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:10.312978 30678 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:10.313050 30855 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:10.313200 31483 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:10.313323 31483 raft_consensus.cc:686] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:10.313565 31483 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:10.315524 30855 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 3.
I20240503 02:13:10.316357 30610 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 2 to 3, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 3 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.550755 30784 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:13:10.553515 31072 consensus_peers.cc:485] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:10.553655 30784 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 02:13:10.554291 31501 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:10.558118 30862 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 02:13:10.559078 31512 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:10.713855 30862 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:13:10.714465 31513 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 02:13:10.717819 30676 consensus_peers.cc:485] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:10.718257 30784 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:13:10.718987 31513 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:10.849721 31447 tablet_copy_client.cc:523] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:10.852968 31447 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:10.892510 31447 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:10.892858 31447 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:10.893092 31447 ts_tablet_manager.cc:1367] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.040s user 0.040s sys 0.000s
I20240503 02:13:10.893796 31447 raft_consensus.cc:348] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:10.894030 31447 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:10.894254 31447 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:10.894796 31447 ts_tablet_manager.cc:1398] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:10.895504 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-60cc6188939e4b728b1c1efea836a7bb received from {username='slave'} at 127.0.0.1:47086
I20240503 02:13:10.895615 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-60cc6188939e4b728b1c1efea836a7bb on tablet 60cc6188939e4b728b1c1efea836a7bb with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:10.910913 31452 tablet_copy_client.cc:523] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:10.913769 31452 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:10.950448 31452 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:10.950904 31452 tablet_bootstrap.cc:492] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:10.951160 31452 ts_tablet_manager.cc:1367] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.037s user 0.032s sys 0.007s
I20240503 02:13:10.951799 31452 raft_consensus.cc:348] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:10.952044 31452 raft_consensus.cc:729] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:10.952226 31452 consensus_queue.cc:260] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:10.952694 31452 ts_tablet_manager.cc:1398] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:13:10.953428 30799 tablet_copy_service.cc:342] P 7a9cbb0303f446618b0e8bc7ad35afb4: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-9b2a8cda34b54f4482afed13d9e43c4f received from {username='slave'} at 127.0.0.1:38744
I20240503 02:13:10.953536 30799 tablet_copy_service.cc:433] P 7a9cbb0303f446618b0e8bc7ad35afb4: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-9b2a8cda34b54f4482afed13d9e43c4f on tablet 9b2a8cda34b54f4482afed13d9e43c4f with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:10.965106 31459 tablet_copy_client.cc:523] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:10.968866 31459 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:10.983309 31465 tablet_copy_client.cc:523] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:10.987130 31465 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:11.011365 31459 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:11.011962 31459 tablet_bootstrap.cc:492] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:11.012233 31459 ts_tablet_manager.cc:1367] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.008s
I20240503 02:13:11.013126 31459 raft_consensus.cc:348] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:11.013429 31459 raft_consensus.cc:729] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:11.013685 31459 consensus_queue.cc:260] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:11.014278 31459 ts_tablet_manager.cc:1398] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:11.015105 31117 tablet_copy_service.cc:342] P 4c1727d9448f4a9c81e2c532e5339801: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-c2e4475d5b86469e90a10f84a910163a received from {username='slave'} at 127.0.0.1:38980
I20240503 02:13:11.015188 31117 tablet_copy_service.cc:433] P 4c1727d9448f4a9c81e2c532e5339801: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-c2e4475d5b86469e90a10f84a910163a on tablet c2e4475d5b86469e90a10f84a910163a with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:11.019809 31470 tablet_copy_client.cc:523] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:11.023674 31470 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:11.032896 31465 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:11.033339 31465 tablet_bootstrap.cc:492] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:11.033823 31465 ts_tablet_manager.cc:1367] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.047s user 0.040s sys 0.006s
I20240503 02:13:11.034842 31475 tablet_copy_client.cc:523] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:11.034842 31465 raft_consensus.cc:348] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:11.035172 31465 raft_consensus.cc:729] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:11.035454 31465 consensus_queue.cc:260] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:11.036149 31465 ts_tablet_manager.cc:1398] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:11.037004 30799 tablet_copy_service.cc:342] P 7a9cbb0303f446618b0e8bc7ad35afb4: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-5277bc5b18a0494baf73181fb4eeba01 received from {username='slave'} at 127.0.0.1:42386
I20240503 02:13:11.037113 30799 tablet_copy_service.cc:433] P 7a9cbb0303f446618b0e8bc7ad35afb4: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-5277bc5b18a0494baf73181fb4eeba01 on tablet 5277bc5b18a0494baf73181fb4eeba01 with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:11.039388 31475 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:11.078068 31470 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:11.078703 31470 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:11.079022 31470 ts_tablet_manager.cc:1367] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.055s user 0.042s sys 0.008s
I20240503 02:13:11.079931 31470 raft_consensus.cc:348] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:11.080299 31470 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:11.080929 31470 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:11.082268 31470 ts_tablet_manager.cc:1398] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:11.083833 30723 tablet_copy_service.cc:342] P ebd1bb9e6d2f4384b781596ce1795b54: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-4e3d6429d7dd499fbe2957fb46fdf30b received from {username='slave'} at 127.0.0.1:48762
I20240503 02:13:11.083953 30723 tablet_copy_service.cc:433] P ebd1bb9e6d2f4384b781596ce1795b54: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-4e3d6429d7dd499fbe2957fb46fdf30b on tablet 4e3d6429d7dd499fbe2957fb46fdf30b with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:11.091295 31475 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:11.091858 31475 tablet_bootstrap.cc:492] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:11.092136 31475 ts_tablet_manager.cc:1367] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.053s user 0.044s sys 0.002s
I20240503 02:13:11.092772 31475 raft_consensus.cc:348] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:11.093039 31475 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:11.093279 31475 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:11.093958 31475 ts_tablet_manager.cc:1398] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:11.094595 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-3f15a775e24c46eda9ad8f5e8a537da4 received from {username='slave'} at 127.0.0.1:39070
I20240503 02:13:11.094748 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-3f15a775e24c46eda9ad8f5e8a537da4 on tablet 3f15a775e24c46eda9ad8f5e8a537da4 with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:11.101778 31256 raft_consensus.cc:3037] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Advancing to term 3
I20240503 02:13:11.104370 31256 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 LEARNER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:11.104902 31512 consensus_queue.cc:1037] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.120607 31512 raft_consensus.cc:1050] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:11.121289 31512 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:11.124270 31256 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 LEARNER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 02:13:11.124742 30784 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 02:13:11.124913 30855 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 02:13:11.124948 31512 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:11.125360 31526 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:11.125746 31525 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:11.128226 31526 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 31 to 34, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.129074 30855 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 31 to 34, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.129164 30784 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 31 to 34, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.134737 30603 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 31 to 34, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.134903 31256 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 31 to 34, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.142163 31106 consensus_queue.cc:237] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:11.144039 30784 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240503 02:13:11.144423 31256 raft_consensus.cc:1261] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240503 02:13:11.144647 31526 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 02:13:11.144942 31524 consensus_queue.cc:1035] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 02:13:11.147173 31526 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.147821 30787 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.148589 31256 raft_consensus.cc:2932] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.152719 30654 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b2a8cda34b54f4482afed13d9e43c4f with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.153434 30610 catalog_manager.cc:5484] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 34 to 35, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 3 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.157745 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2a8cda34b54f4482afed13d9e43c4f with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:11.158780 31530 tablet_replica.cc:330] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:11.158975 31530 raft_consensus.cc:2229] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.159345 31530 raft_consensus.cc:2258] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.160216 31530 ts_tablet_manager.cc:1875] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.165675 31530 ts_tablet_manager.cc:1888] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.34
I20240503 02:13:11.165859 31530 log.cc:1196] T 9b2a8cda34b54f4482afed13d9e43c4f P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/9b2a8cda34b54f4482afed13d9e43c4f
I20240503 02:13:11.166730 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet 9b2a8cda34b54f4482afed13d9e43c4f (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:11.179728 31182 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 LEARNER]: Advancing to term 3
I20240503 02:13:11.182590 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEARNER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:11.183199 31513 consensus_queue.cc:1037] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:11.186421 31513 raft_consensus.cc:1050] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:11.187312 31513 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.189733 30855 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:13:11.190255 31484 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.190564 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEARNER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:13:11.191074 31484 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.191287 30787 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:13:11.192312 31484 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.194775 31483 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.195430 31182 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.195771 30787 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.200335 30605 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 28 to 31, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.203728 31106 raft_consensus.cc:1203] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 02:13:11.205345 30862 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.214085 30707 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.215720 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:13:11.215776 30862 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:13:11.216233 31513 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.216475 31484 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.218992 31514 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.219614 30855 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.221788 31182 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.223685 30597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e3d6429d7dd499fbe2957fb46fdf30b with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.224885 30603 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 31 to 32, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 3 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.229204 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3d6429d7dd499fbe2957fb46fdf30b with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:11.235251 31533 tablet_replica.cc:330] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:11.235536 31533 raft_consensus.cc:2229] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.235987 31533 raft_consensus.cc:2258] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.237340 31533 ts_tablet_manager.cc:1875] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.243214 31533 ts_tablet_manager.cc:1888] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240503 02:13:11.243347 31533 log.cc:1196] T 4e3d6429d7dd499fbe2957fb46fdf30b P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/4e3d6429d7dd499fbe2957fb46fdf30b
I20240503 02:13:11.244105 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet 4e3d6429d7dd499fbe2957fb46fdf30b (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:11.246336 31182 raft_consensus.cc:1203] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20240503 02:13:11.350033 31256 raft_consensus.cc:1203] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20240503 02:13:11.401257 31106 raft_consensus.cc:1203] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 02:13:11.487615 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:11.491465 30707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:11.491803 30707 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Received request to transfer leadership to TS d7b4ee58dad047fe8c54f6a1ae18f0c4
I20240503 02:13:11.493006 30707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d906796832b043e1ae7fe4825491a008"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:11.493247 30707 raft_consensus.cc:595] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:11.493716 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:13:11.493963 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:11.599187 31534 raft_consensus.cc:1050] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:11.600171 31534 consensus_queue.cc:237] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:11.603392 31106 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.603863 30862 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.603992 30707 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.603874 31536 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.604478 31534 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.605329 31502 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.617374 31534 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.618351 30707 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.618387 30862 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.623035 30609 catalog_manager.cc:5484] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.623762 31106 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.628717 30784 consensus_queue.cc:237] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:11.630726 30707 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:11.631260 31534 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.631402 31106 raft_consensus.cc:1261] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:11.631933 31534 consensus_queue.cc:1035] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.633466 31502 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.634639 31106 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.637054 30707 raft_consensus.cc:2932] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.639446 30596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5277bc5b18a0494baf73181fb4eeba01 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.640625 30609 catalog_manager.cc:5484] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.644325 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet 5277bc5b18a0494baf73181fb4eeba01 with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:11.644641 31530 tablet_replica.cc:330] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:11.644834 31530 raft_consensus.cc:2229] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.645009 31530 raft_consensus.cc:2258] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.647193 31530 ts_tablet_manager.cc:1875] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.651777 31530 ts_tablet_manager.cc:1888] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 02:13:11.651916 31530 log.cc:1196] T 5277bc5b18a0494baf73181fb4eeba01 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/5277bc5b18a0494baf73181fb4eeba01
I20240503 02:13:11.652526 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet 5277bc5b18a0494baf73181fb4eeba01 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:11.670480 31526 raft_consensus.cc:1050] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:11.671162 31526 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.673256 30862 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:11.673537 31182 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:11.673996 31512 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.674072 30784 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:11.674335 31526 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.674888 31512 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.678701 31526 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.679395 30784 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.683437 31182 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.684170 30605 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 31 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.691484 30862 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.694787 31107 consensus_queue.cc:237] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.696961 31182 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:11.696998 30862 raft_consensus.cc:1261] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 4c1727d9448f4a9c81e2c532e5339801: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:11.698611 31512 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:11.699005 31540 consensus_queue.cc:1035] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:11.699492 31514 raft_consensus.cc:979] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:11.699617 31514 raft_consensus.cc:1067] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:11.700392 30784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d906796832b043e1ae7fe4825491a008"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42342
I20240503 02:13:11.700717 30784 raft_consensus.cc:482] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:11.701325 30784 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:11.701540 31540 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.702881 31182 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.703644 30855 raft_consensus.cc:2932] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.705010 30784 raft_consensus.cc:504] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:11.706132 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:11.706143 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:11.706377 30707 raft_consensus.cc:3032] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:11.706440 30707 raft_consensus.cc:729] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:11.706490 30861 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:11.706661 30707 consensus_queue.cc:260] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:11.706982 30707 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:11.709033 30707 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:11.709376 30861 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 2.
I20240503 02:13:11.710945 30752 leader_election.cc:302] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:11.711218 31534 raft_consensus.cc:2781] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:11.711339 31534 raft_consensus.cc:686] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:11.711647 31534 consensus_queue.cc:237] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:11.711652 30605 catalog_manager.cc:5484] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 reported cstate change: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 2 leader_uuid: "4c1727d9448f4a9c81e2c532e5339801" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:11.713583 30654 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2e4475d5b86469e90a10f84a910163a with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.709527 30784 leader_election.cc:288] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 2 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:11.715044 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e4475d5b86469e90a10f84a910163a with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:11.715356 31533 tablet_replica.cc:330] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:11.715531 31533 raft_consensus.cc:2229] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.715790 31533 raft_consensus.cc:2258] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.716657 31533 ts_tablet_manager.cc:1875] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.717274 30603 catalog_manager.cc:5484] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 1 to 2, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.723337 31533 ts_tablet_manager.cc:1888] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20240503 02:13:11.723481 31533 log.cc:1196] T c2e4475d5b86469e90a10f84a910163a P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/c2e4475d5b86469e90a10f84a910163a
I20240503 02:13:11.724153 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet c2e4475d5b86469e90a10f84a910163a (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:11.742353 31514 raft_consensus.cc:979] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:11.742559 31483 raft_consensus.cc:1067] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Signalling peer d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:11.743108 30861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:47064
I20240503 02:13:11.743294 30861 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:11.743382 30861 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:11.746050 30861 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.746737 30861 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 4 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601)
I20240503 02:13:11.747236 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
I20240503 02:13:11.747447 31182 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:11.747557 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:11.747711 30707 raft_consensus.cc:3032] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:13:11.747782 30707 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:11.748034 30707 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.748392 30707 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:11.750540 31182 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 4.
I20240503 02:13:11.751044 30826 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf667d67d5b341c7af3b984e0c1f2d55, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:11.751343 31480 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:13:11.751554 30707 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 4.
I20240503 02:13:11.751771 31480 raft_consensus.cc:686] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:11.752065 31480 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:11.754618 30603 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 3 to 4, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.904078 31547 raft_consensus.cc:1050] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:11.904867 31547 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:11.907125 31256 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:13:11.907182 31182 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:13:11.907656 31546 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.908053 30707 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:13:11.908255 31547 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.908612 31546 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.912245 31547 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.912959 31480 raft_consensus.cc:1050] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:11.912796 31256 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.913825 31480 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:11.913785 30707 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.919427 30603 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 30 to 31, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.924147 31182 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.926440 30707 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.927031 31547 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.927148 30784 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.927335 31106 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:11.927888 31547 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.929947 31554 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:11.933053 31554 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.936784 30784 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.937448 30861 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:11.939406 31182 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 02:13:11.939406 31256 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 02:13:11.940366 31554 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.940793 31547 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:11.943291 31552 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.942458 30707 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.944044 31181 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.947351 31256 raft_consensus.cc:2932] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.947515 30605 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.948490 30610 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 reported cstate change: config changed from index 31 to 32, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 1 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:11.949638 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f15a775e24c46eda9ad8f5e8a537da4 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.955801 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f15a775e24c46eda9ad8f5e8a537da4 with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:11.957098 31558 tablet_replica.cc:330] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:11.957418 31558 raft_consensus.cc:2229] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.957778 31558 raft_consensus.cc:2258] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.958835 31558 ts_tablet_manager.cc:1875] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.959369 31106 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.964525 30855 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:11.966575 31558 ts_tablet_manager.cc:1888] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20240503 02:13:11.966933 31558 log.cc:1196] T 3f15a775e24c46eda9ad8f5e8a537da4 P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/3f15a775e24c46eda9ad8f5e8a537da4
I20240503 02:13:11.967028 31106 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:11.967209 30784 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:11.967821 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet 3f15a775e24c46eda9ad8f5e8a537da4 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:11.967857 31552 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.968246 31554 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:11.970418 31480 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.971163 30784 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.971887 31106 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.975498 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60cc6188939e4b728b1c1efea836a7bb with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.976629 30605 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 30 to 31, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:11.980209 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cc6188939e4b728b1c1efea836a7bb with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:11.980563 31558 tablet_replica.cc:330] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:11.980940 31558 raft_consensus.cc:2229] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.981194 31558 raft_consensus.cc:2258] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.982206 31558 ts_tablet_manager.cc:1875] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.987120 31558 ts_tablet_manager.cc:1888] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 02:13:11.987233 31558 log.cc:1196] T 60cc6188939e4b728b1c1efea836a7bb P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/60cc6188939e4b728b1c1efea836a7bb
I20240503 02:13:11.987879 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet 60cc6188939e4b728b1c1efea836a7bb (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:12.132154 30707 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:12.132869 31534 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:12.136741 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 02:13:12.137341 31480 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:12.138584 30855 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:13:12.139437 31502 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:13:12.144500 30712 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 02:13:12.145102 31552 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:12.983398 30855 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:12.985437 30784 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:12.985472 31106 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:12.985920 31566 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:12.986193 31567 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240503 02:13:12.987392 30829 consensus_peers.cc:485] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60cc6188939e4b728b1c1efea836a7bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.988776 31566 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:12.992292 31106 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:12.996430 30787 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:12.996431 30606 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 31 to 32, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.999900 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:12.999912 30787 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:13.003063 31534 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:13.005874 31502 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:13.006404 30712 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:13.006302 31256 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
W20240503 02:13:13.010102 30752 consensus_peers.cc:485] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8c3012068484fe3acd108cc36eb62d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.011217 31536 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:13.012387 30609 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 30 to 32, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:13.029989 30712 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:13.040449 31256 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:13.043326 30712 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:13.044028 31523 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:13.048010 30855 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:13:13.048846 31582 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240503 02:13:13.052034 31219 consensus_peers.cc:485] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c81f82e2608a401ba8f4f381c3ad95be. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.051867 31523 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.052569 30712 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.053736 30855 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.058414 30605 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 30 to 32, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 2 leader_uuid: "3e8a84e16aa2470899683bc9edef9283" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.059697 30855 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:13.061270 30712 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
W20240503 02:13:13.061591 30829 consensus_peers.cc:485] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 -> Peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209): Couldn't send request to peer 3e8a84e16aa2470899683bc9edef9283. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e3d6429d7dd499fbe2957fb46fdf30b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.061709 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 02:13:13.062013 31567 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:13.062276 31566 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:13.064450 31566 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 32 to 34, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:13.066303 31182 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:13.067095 30712 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } } }
I20240503 02:13:13.069056 30609 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 32 to 34, NON_VOTER 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) added. New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.071660 30707 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:13.075299 30855 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:13.075299 30787 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20240503 02:13:13.075601 30676 consensus_peers.cc:485] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601): Couldn't send request to peer cf667d67d5b341c7af3b984e0c1f2d55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 106b5308c1b34e119da27c3c75178ecb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.076709 31590 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:13.077061 31483 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:13.079725 31590 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.080242 30855 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.083164 30787 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } } }
I20240503 02:13:13.085477 30610 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index -1 to 29, NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) added. New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.088274 30707 consensus_queue.cc:237] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:13.093333 30787 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:13.093963 31593 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:13.098026 31590 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:13.098845 30787 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
I20240503 02:13:13.105424 30609 catalog_manager.cc:5484] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.109144 30855 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:13:13.111270 31590 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:13.114174 30855 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } } }
W20240503 02:13:13.119325 30677 consensus_peers.cc:485] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 -> Peer 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68:40457): Couldn't send request to peer 4c1727d9448f4a9c81e2c532e5339801. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 441b8fdd54574e0f837d04e0916d1bf1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:13.443678 31600 ts_tablet_manager.cc:902] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:13.447912 31600 tablet_copy_client.cc:308] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.66:46225
I20240503 02:13:13.448802 30799 tablet_copy_service.cc:140] P 7a9cbb0303f446618b0e8bc7ad35afb4: Received BeginTabletCopySession request for tablet c8c3012068484fe3acd108cc36eb62d5 from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:42386)
I20240503 02:13:13.448978 30799 tablet_copy_service.cc:161] P 7a9cbb0303f446618b0e8bc7ad35afb4: Beginning new tablet copy session on tablet c8c3012068484fe3acd108cc36eb62d5 from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:42386: session id = 4c1727d9448f4a9c81e2c532e5339801-c8c3012068484fe3acd108cc36eb62d5
I20240503 02:13:13.453382 30799 tablet_copy_source_session.cc:205] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.454608 31605 ts_tablet_manager.cc:902] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209)
I20240503 02:13:13.455106 31600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c3012068484fe3acd108cc36eb62d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.467242 31607 ts_tablet_manager.cc:902] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:13.467636 31600 tablet_copy_client.cc:783] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.467945 31600 tablet_copy_client.cc:647] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.469712 31604 tablet_copy_client.cc:1002] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.483209 31607 tablet_copy_client.cc:308] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:13.484093 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet 4e3d6429d7dd499fbe2957fb46fdf30b from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:39070)
I20240503 02:13:13.484249 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet 4e3d6429d7dd499fbe2957fb46fdf30b from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:39070: session id = 3e8a84e16aa2470899683bc9edef9283-4e3d6429d7dd499fbe2957fb46fdf30b
I20240503 02:13:13.485877 31605 tablet_copy_client.cc:308] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.70:46209
I20240503 02:13:13.487694 30873 tablet_copy_source_session.cc:205] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.490525 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3d6429d7dd499fbe2957fb46fdf30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.494338 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:13.495335 31266 tablet_copy_service.cc:140] P 3e8a84e16aa2470899683bc9edef9283: Received BeginTabletCopySession request for tablet c81f82e2608a401ba8f4f381c3ad95be from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:35958)
I20240503 02:13:13.495633 31266 tablet_copy_service.cc:161] P 3e8a84e16aa2470899683bc9edef9283: Beginning new tablet copy session on tablet c81f82e2608a401ba8f4f381c3ad95be from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:35958: session id = cf667d67d5b341c7af3b984e0c1f2d55-c81f82e2608a401ba8f4f381c3ad95be
I20240503 02:13:13.498795 30855 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
new_leader_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
from {username='slave'} at 127.0.0.1:47090
I20240503 02:13:13.499002 31266 tablet_copy_source_session.cc:205] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.499034 30855 raft_consensus.cc:595] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Received request to transfer leadership to TS cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:13.500295 31605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81f82e2608a401ba8f4f381c3ad95be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.500926 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d906796832b043e1ae7fe4825491a008"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
new_leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:42406
I20240503 02:13:13.501130 30787 raft_consensus.cc:595] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Received request to transfer leadership to TS ebd1bb9e6d2f4384b781596ce1795b54
I20240503 02:13:13.501701 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:13:13.501971 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:13.503508 31607 tablet_copy_client.cc:783] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.503728 31607 tablet_copy_client.cc:647] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.505265 31612 tablet_copy_client.cc:1002] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.508589 31605 tablet_copy_client.cc:783] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.508841 31605 tablet_copy_client.cc:647] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.510946 31614 tablet_copy_client.cc:1002] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.525290 31616 ts_tablet_manager.cc:902] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Initiating tablet copy from peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:13.542656 31616 tablet_copy_client.cc:308] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.65:40107
I20240503 02:13:13.543354 30723 tablet_copy_service.cc:140] P ebd1bb9e6d2f4384b781596ce1795b54: Received BeginTabletCopySession request for tablet 106b5308c1b34e119da27c3c75178ecb from peer cf667d67d5b341c7af3b984e0c1f2d55 ({username='slave'} at 127.0.0.1:48762)
I20240503 02:13:13.543481 30723 tablet_copy_service.cc:161] P ebd1bb9e6d2f4384b781596ce1795b54: Beginning new tablet copy session on tablet 106b5308c1b34e119da27c3c75178ecb from peer cf667d67d5b341c7af3b984e0c1f2d55 at {username='slave'} at 127.0.0.1:48762: session id = cf667d67d5b341c7af3b984e0c1f2d55-106b5308c1b34e119da27c3c75178ecb
I20240503 02:13:13.546563 30723 tablet_copy_source_session.cc:205] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.547828 31616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106b5308c1b34e119da27c3c75178ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.554112 31616 tablet_copy_client.cc:783] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.554394 31616 tablet_copy_client.cc:647] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.556196 31620 tablet_copy_client.cc:1002] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.559177 31621 ts_tablet_manager.cc:902] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Initiating tablet copy from peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:13.568822 31621 tablet_copy_client.cc:308] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.67:44143
I20240503 02:13:13.569506 30873 tablet_copy_service.cc:140] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Received BeginTabletCopySession request for tablet 60cc6188939e4b728b1c1efea836a7bb from peer 3e8a84e16aa2470899683bc9edef9283 ({username='slave'} at 127.0.0.1:39070)
I20240503 02:13:13.569684 30873 tablet_copy_service.cc:161] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Beginning new tablet copy session on tablet 60cc6188939e4b728b1c1efea836a7bb from peer 3e8a84e16aa2470899683bc9edef9283 at {username='slave'} at 127.0.0.1:39070: session id = 3e8a84e16aa2470899683bc9edef9283-60cc6188939e4b728b1c1efea836a7bb
I20240503 02:13:13.574127 30873 tablet_copy_source_session.cc:205] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.575371 31621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cc6188939e4b728b1c1efea836a7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.582442 31621 tablet_copy_client.cc:783] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.582727 31621 tablet_copy_client.cc:647] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.584491 31626 tablet_copy_client.cc:1002] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.609215 31534 raft_consensus.cc:979] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:13.609386 31534 raft_consensus.cc:1067] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Signalling peer ebd1bb9e6d2f4384b781596ce1795b54 to start an election
I20240503 02:13:13.609977 30707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d906796832b043e1ae7fe4825491a008"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
from {username='slave'} at 127.0.0.1:60496
I20240503 02:13:13.610144 30707 raft_consensus.cc:482] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:13.610265 30707 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:13.612810 30707 raft_consensus.cc:504] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:13.613734 30855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:13.613734 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d906796832b043e1ae7fe4825491a008" candidate_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:13.613989 30855 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:13.613998 30787 raft_consensus.cc:3032] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:13.614113 30787 raft_consensus.cc:729] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:13.614349 30787 consensus_queue.cc:260] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:13.614709 30787 raft_consensus.cc:3037] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:13.613772 30707 leader_election.cc:288] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 3 election: Requested vote from peers 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:13.617143 30855 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 3.
I20240503 02:13:13.617535 30787 raft_consensus.cc:2454] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd1bb9e6d2f4384b781596ce1795b54 in term 3.
I20240503 02:13:13.617621 30677 leader_election.cc:302] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b4ee58dad047fe8c54f6a1ae18f0c4, ebd1bb9e6d2f4384b781596ce1795b54; no voters:
I20240503 02:13:13.617914 31593 raft_consensus.cc:2781] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:13.618057 31593 raft_consensus.cc:686] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Becoming Leader. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:13.618305 31593 consensus_queue.cc:237] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } }
I20240503 02:13:13.621501 30605 catalog_manager.cc:5484] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: term changed from 2 to 3, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65). New cstate: current_term: 3 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.644717 31631 ts_tablet_manager.cc:902] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Initiating tablet copy from peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107)
I20240503 02:13:13.646880 31631 tablet_copy_client.cc:308] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Beginning tablet copy session from remote peer at address 127.26.178.65:40107
I20240503 02:13:13.656409 30723 tablet_copy_service.cc:140] P ebd1bb9e6d2f4384b781596ce1795b54: Received BeginTabletCopySession request for tablet 441b8fdd54574e0f837d04e0916d1bf1 from peer 4c1727d9448f4a9c81e2c532e5339801 ({username='slave'} at 127.0.0.1:48766)
I20240503 02:13:13.656596 30723 tablet_copy_service.cc:161] P ebd1bb9e6d2f4384b781596ce1795b54: Beginning new tablet copy session on tablet 441b8fdd54574e0f837d04e0916d1bf1 from peer 4c1727d9448f4a9c81e2c532e5339801 at {username='slave'} at 127.0.0.1:48766: session id = 4c1727d9448f4a9c81e2c532e5339801-441b8fdd54574e0f837d04e0916d1bf1
I20240503 02:13:13.659031 30723 tablet_copy_source_session.cc:205] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.660228 31631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441b8fdd54574e0f837d04e0916d1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.665940 31631 tablet_copy_client.cc:783] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.666168 31631 tablet_copy_client.cc:647] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.668778 31635 tablet_copy_client.cc:1002] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Injecting latency into file download: 2000
I20240503 02:13:13.962649 31638 raft_consensus.cc:979] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: : Instructing follower cf667d67d5b341c7af3b984e0c1f2d55 to start an election
I20240503 02:13:13.962831 31638 raft_consensus.cc:1067] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Signalling peer cf667d67d5b341c7af3b984e0c1f2d55 to start an election
I20240503 02:13:13.963405 31182 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4"
dest_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
from {username='slave'} at 127.0.0.1:57056
I20240503 02:13:13.963608 31182 raft_consensus.cc:482] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:13.963732 31182 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:13.966226 31182 raft_consensus.cc:504] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:13.967525 31182 leader_election.cc:288] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [CANDIDATE]: Term 2 election: Requested vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70:46209)
I20240503 02:13:13.967594 31256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "3e8a84e16aa2470899683bc9edef9283"
I20240503 02:13:13.967757 31256 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:13.968065 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f15a775e24c46eda9ad8f5e8a537da4" candidate_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:13.968243 30861 raft_consensus.cc:3032] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:13.968307 30861 raft_consensus.cc:729] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:13.968572 30861 consensus_queue.cc:260] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:13.969033 30861 raft_consensus.cc:3037] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:13.970943 31256 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf667d67d5b341c7af3b984e0c1f2d55 in term 2.
I20240503 02:13:13.971365 31148 leader_election.cc:302] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a84e16aa2470899683bc9edef9283, cf667d67d5b341c7af3b984e0c1f2d55; no voters:
I20240503 02:13:13.971654 31495 raft_consensus.cc:2781] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:13.972445 30861 raft_consensus.cc:2454] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf667d67d5b341c7af3b984e0c1f2d55 in term 2.
I20240503 02:13:13.973057 31495 raft_consensus.cc:686] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEADER]: Becoming Leader. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Running, Role: LEADER
I20240503 02:13:13.973341 31495 consensus_queue.cc:237] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:13.976509 30605 catalog_manager.cc:5484] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 reported cstate change: term changed from 1 to 2, leader changed from d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) to cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69). New cstate: current_term: 2 leader_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:14.079480 30787 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 02:13:14.080039 31593 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:14.083763 30855 raft_consensus.cc:1261] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 02:13:14.084717 31593 consensus_queue.cc:1035] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:13:14.478749 30855 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer cf667d67d5b341c7af3b984e0c1f2d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:14.479772 31255 raft_consensus.cc:1261] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer cf667d67d5b341c7af3b984e0c1f2d55: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:14.479866 31495 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:14.483669 31650 consensus_queue.cc:1035] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:15.470537 31600 tablet_copy_client.cc:523] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.474017 31600 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:15.502296 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:15.505002 30707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
mode: GRACEFUL
new_leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:60514
I20240503 02:13:15.505234 30707 raft_consensus.cc:595] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Received request to transfer leadership to TS 7a9cbb0303f446618b0e8bc7ad35afb4
I20240503 02:13:15.505589 31607 tablet_copy_client.cc:523] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.508982 31607 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:15.511307 31605 tablet_copy_client.cc:523] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.511373 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 02:13:15.511631 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:15.512889 31600 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.513156 31600 tablet_bootstrap.cc:492] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:15.513339 31600 ts_tablet_manager.cc:1367] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.039s user 0.033s sys 0.005s
I20240503 02:13:15.513945 31600 raft_consensus.cc:348] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:15.514377 31600 raft_consensus.cc:729] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:15.514566 31600 consensus_queue.cc:260] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:15.515041 31605 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:15.516381 31600 ts_tablet_manager.cc:1398] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.003s user 0.004s sys 0.001s
I20240503 02:13:15.516964 30799 tablet_copy_service.cc:342] P 7a9cbb0303f446618b0e8bc7ad35afb4: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-c8c3012068484fe3acd108cc36eb62d5 received from {username='slave'} at 127.0.0.1:42386
I20240503 02:13:15.517056 30799 tablet_copy_service.cc:433] P 7a9cbb0303f446618b0e8bc7ad35afb4: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-c8c3012068484fe3acd108cc36eb62d5 on tablet c8c3012068484fe3acd108cc36eb62d5 with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:15.556489 31607 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.556540 31616 tablet_copy_client.cc:523] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.557011 31607 tablet_bootstrap.cc:492] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:15.557281 31607 ts_tablet_manager.cc:1367] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.048s user 0.040s sys 0.007s
I20240503 02:13:15.558055 31607 raft_consensus.cc:348] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:15.558286 31607 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:15.558483 31607 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:15.559613 31607 ts_tablet_manager.cc:1398] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:15.560364 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-4e3d6429d7dd499fbe2957fb46fdf30b received from {username='slave'} at 127.0.0.1:39070
I20240503 02:13:15.560472 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-4e3d6429d7dd499fbe2957fb46fdf30b on tablet 4e3d6429d7dd499fbe2957fb46fdf30b with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:15.561386 31616 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap starting.
I20240503 02:13:15.564882 31659 raft_consensus.cc:979] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: : Instructing follower 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:15.565104 31593 raft_consensus.cc:1067] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Signalling peer 7a9cbb0303f446618b0e8bc7ad35afb4 to start an election
I20240503 02:13:15.565747 30784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
from {username='slave'} at 127.0.0.1:42342
I20240503 02:13:15.565948 30784 raft_consensus.cc:482] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:15.566061 30784 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:15.566071 31605 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.566471 31605 tablet_bootstrap.cc:492] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:15.566787 31605 ts_tablet_manager.cc:1367] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.052s user 0.045s sys 0.004s
I20240503 02:13:15.567541 31605 raft_consensus.cc:348] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.567873 31605 raft_consensus.cc:729] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:15.568145 31605 consensus_queue.cc:260] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.568900 31605 ts_tablet_manager.cc:1398] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:15.569202 30784 raft_consensus.cc:504] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.569553 31266 tablet_copy_service.cc:342] P 3e8a84e16aa2470899683bc9edef9283: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-c81f82e2608a401ba8f4f381c3ad95be received from {username='slave'} at 127.0.0.1:35958
I20240503 02:13:15.569761 31266 tablet_copy_service.cc:433] P 3e8a84e16aa2470899683bc9edef9283: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-c81f82e2608a401ba8f4f381c3ad95be on tablet c81f82e2608a401ba8f4f381c3ad95be with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:15.570832 30784 leader_election.cc:288] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 3 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143)
I20240503 02:13:15.571491 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:15.571635 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:15.571705 30861 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:15.571812 30707 raft_consensus.cc:3032] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:13:15.571892 30707 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ebd1bb9e6d2f4384b781596ce1795b54, State: Running, Role: LEADER
I20240503 02:13:15.572171 30707 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.572607 30707 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:15.574517 30861 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 3.
I20240503 02:13:15.574935 30752 leader_election.cc:302] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters:
I20240503 02:13:15.575392 31640 raft_consensus.cc:2781] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:15.576332 30707 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9cbb0303f446618b0e8bc7ad35afb4 in term 3.
I20240503 02:13:15.576592 31640 raft_consensus.cc:686] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Becoming Leader. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:15.576925 31640 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.579835 30605 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: term changed from 2 to 3, leader changed from ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) to 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66). New cstate: current_term: 3 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.584900 31621 tablet_copy_client.cc:523] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.588996 31621 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Bootstrap starting.
I20240503 02:13:15.595764 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:15.596012 30784 raft_consensus.cc:595] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Received request to transfer leadership
I20240503 02:13:15.604907 31616 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.605356 31616 tablet_bootstrap.cc:492] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Bootstrap complete.
I20240503 02:13:15.605681 31616 ts_tablet_manager.cc:1367] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Time spent bootstrapping tablet: real 0.046s user 0.029s sys 0.011s
I20240503 02:13:15.605747 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:15.605978 30784 raft_consensus.cc:595] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Received request to transfer leadership
W20240503 02:13:15.606357 30596 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 106b5308c1b34e119da27c3c75178ecb already in progress [suppressed 1 similar messages]
I20240503 02:13:15.606470 31616 raft_consensus.cc:348] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.606765 31616 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf667d67d5b341c7af3b984e0c1f2d55, State: Initialized, Role: LEARNER
I20240503 02:13:15.607044 31616 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:15.608696 31616 ts_tablet_manager.cc:1398] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 02:13:15.609786 30723 tablet_copy_service.cc:342] P ebd1bb9e6d2f4384b781596ce1795b54: Request end of tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-106b5308c1b34e119da27c3c75178ecb received from {username='slave'} at 127.0.0.1:48762
I20240503 02:13:15.609930 30723 tablet_copy_service.cc:433] P ebd1bb9e6d2f4384b781596ce1795b54: ending tablet copy session cf667d67d5b341c7af3b984e0c1f2d55-106b5308c1b34e119da27c3c75178ecb on tablet 106b5308c1b34e119da27c3c75178ecb with peer cf667d67d5b341c7af3b984e0c1f2d55
I20240503 02:13:15.618621 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:15.618903 30784 raft_consensus.cc:595] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Received request to transfer leadership
W20240503 02:13:15.619558 30659 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 106b5308c1b34e119da27c3c75178ecb already in progress
W20240503 02:13:15.619639 30659 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 106b5308c1b34e119da27c3c75178ecb already in progress
I20240503 02:13:15.628451 31621 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.628898 31621 tablet_bootstrap.cc:492] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Bootstrap complete.
I20240503 02:13:15.629202 31621 ts_tablet_manager.cc:1367] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Time spent bootstrapping tablet: real 0.040s user 0.037s sys 0.001s
I20240503 02:13:15.629966 31621 raft_consensus.cc:348] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:15.630230 31621 raft_consensus.cc:729] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Initialized, Role: LEARNER
I20240503 02:13:15.630476 31621 consensus_queue.cc:260] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: NON_VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: true } }
I20240503 02:13:15.631093 31621 ts_tablet_manager.cc:1398] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:15.631745 30873 tablet_copy_service.cc:342] P d7b4ee58dad047fe8c54f6a1ae18f0c4: Request end of tablet copy session 3e8a84e16aa2470899683bc9edef9283-60cc6188939e4b728b1c1efea836a7bb received from {username='slave'} at 127.0.0.1:39070
I20240503 02:13:15.631882 30873 tablet_copy_service.cc:433] P d7b4ee58dad047fe8c54f6a1ae18f0c4: ending tablet copy session 3e8a84e16aa2470899683bc9edef9283-60cc6188939e4b728b1c1efea836a7bb on tablet 60cc6188939e4b728b1c1efea836a7bb with peer 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:15.669131 31631 tablet_copy_client.cc:523] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:15.671952 31631 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap starting.
I20240503 02:13:15.704376 31631 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:15.704797 31631 tablet_bootstrap.cc:492] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Bootstrap complete.
I20240503 02:13:15.705090 31631 ts_tablet_manager.cc:1367] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.006s
I20240503 02:13:15.705754 31631 raft_consensus.cc:348] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:15.705953 31631 raft_consensus.cc:729] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4c1727d9448f4a9c81e2c532e5339801, State: Initialized, Role: LEARNER
I20240503 02:13:15.706148 31631 consensus_queue.cc:260] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: true } }
I20240503 02:13:15.706610 31631 ts_tablet_manager.cc:1398] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:15.707267 30723 tablet_copy_service.cc:342] P ebd1bb9e6d2f4384b781596ce1795b54: Request end of tablet copy session 4c1727d9448f4a9c81e2c532e5339801-441b8fdd54574e0f837d04e0916d1bf1 received from {username='slave'} at 127.0.0.1:48766
I20240503 02:13:15.707420 30723 tablet_copy_service.cc:433] P ebd1bb9e6d2f4384b781596ce1795b54: ending tablet copy session 4c1727d9448f4a9c81e2c532e5339801-441b8fdd54574e0f837d04e0916d1bf1 on tablet 441b8fdd54574e0f837d04e0916d1bf1 with peer 4c1727d9448f4a9c81e2c532e5339801
I20240503 02:13:15.736994 31106 raft_consensus.cc:1203] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240503 02:13:15.737854 31106 raft_consensus.cc:1203] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 02:13:15.884231 31182 raft_consensus.cc:1203] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240503 02:13:15.997107 31255 raft_consensus.cc:1203] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.33->[4.34-4.34] Dedup: 4.34->[]
I20240503 02:13:16.012840 31182 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Advancing to term 3
I20240503 02:13:16.015421 31182 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEARNER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:13:16.016012 31640 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.020375 31640 raft_consensus.cc:1050] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
W20240503 02:13:16.020551 31640 raft_consensus.cc:1054] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Unable to promote non-voter cf667d67d5b341c7af3b984e0c1f2d55: Service unavailable: leader transfer in progress
I20240503 02:13:16.082904 31255 raft_consensus.cc:1203] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240503 02:13:16.088514 30861 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:13:16.089121 31661 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.089723 30712 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:13:16.090227 31661 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.091750 31640 raft_consensus.cc:979] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: : Instructing follower d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:16.091977 31661 raft_consensus.cc:1067] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Signalling peer d7b4ee58dad047fe8c54f6a1ae18f0c4 to start an election
I20240503 02:13:16.092882 30855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
from {username='slave'} at 127.0.0.1:47060
I20240503 02:13:16.093847 30855 raft_consensus.cc:482] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:16.094013 30855 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:16.097287 30855 raft_consensus.cc:504] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:16.099349 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4"
I20240503 02:13:16.099603 30787 raft_consensus.cc:3032] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:13:16.099711 30787 raft_consensus.cc:729] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a9cbb0303f446618b0e8bc7ad35afb4, State: Running, Role: LEADER
I20240503 02:13:16.100008 30787 consensus_queue.cc:260] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:16.100500 30787 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:16.100553 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106b5308c1b34e119da27c3c75178ecb" candidate_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "ebd1bb9e6d2f4384b781596ce1795b54"
I20240503 02:13:16.104517 30787 raft_consensus.cc:2454] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b4ee58dad047fe8c54f6a1ae18f0c4 in term 4.
I20240503 02:13:16.104815 30855 leader_election.cc:288] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 4 election: Requested vote from peers ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107), 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225)
I20240503 02:13:16.104993 30828 leader_election.cc:302] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9cbb0303f446618b0e8bc7ad35afb4, d7b4ee58dad047fe8c54f6a1ae18f0c4; no voters: ebd1bb9e6d2f4384b781596ce1795b54
I20240503 02:13:16.105505 31552 raft_consensus.cc:2781] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:13:16.106191 31552 raft_consensus.cc:686] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Becoming Leader. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:16.106492 31552 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } }
I20240503 02:13:16.108875 30605 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: term changed from 3 to 4, leader changed from 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) to d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67). New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:16.270852 31681 raft_consensus.cc:1050] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:16.271685 31681 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:16.274430 30861 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.274960 31588 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.275101 30712 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.275326 31182 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEARNER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.276110 31588 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.276587 31588 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.277649 31683 raft_consensus.cc:1050] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:16.278429 31683 consensus_queue.cc:237] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:16.282416 30861 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:16.282949 31106 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:16.283144 31593 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.286132 31593 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.293776 30707 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.293879 31588 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.301247 30855 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.301548 30605 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 32 to 33, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3e8a84e16aa2470899683bc9edef9283" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.306497 31182 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.307265 30787 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:13:16.307802 31683 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.308944 31684 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.311273 31255 consensus_queue.cc:237] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:16.313308 30855 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.314159 30605 catalog_manager.cc:5484] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:16.314484 31181 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.315376 31106 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.317245 31682 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:16.317281 30787 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } attrs { replace: true } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.320292 30712 raft_consensus.cc:1261] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.320801 31682 consensus_queue.cc:1035] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:16.322101 31682 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.322911 31182 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.324141 30712 consensus_queue.cc:237] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:16.324365 30707 raft_consensus.cc:2932] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.326119 30658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c81f82e2608a401ba8f4f381c3ad95be with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.326871 31106 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:16.327391 31683 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:16.329147 30605 catalog_manager.cc:5484] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 33 to 34, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 2 leader_uuid: "3e8a84e16aa2470899683bc9edef9283" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:16.330049 30784 raft_consensus.cc:1261] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer ebd1bb9e6d2f4384b781596ce1795b54: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:13:16.331324 31593 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.331872 31107 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.334407 30597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 441b8fdd54574e0f837d04e0916d1bf1 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.335014 31593 consensus_queue.cc:1035] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:16.335228 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet c81f82e2608a401ba8f4f381c3ad95be with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:16.335934 30605 catalog_manager.cc:5484] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New cstate: current_term: 2 leader_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.337623 31695 tablet_replica.cc:330] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:16.337944 31695 raft_consensus.cc:2229] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.338155 31695 raft_consensus.cc:2258] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.338999 30787 raft_consensus.cc:2932] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.340255 30852 tablet_service.cc:1514] Processing DeleteTablet for tablet 441b8fdd54574e0f837d04e0916d1bf1 with delete_type TABLET_DATA_TOMBSTONED (TS d7b4ee58dad047fe8c54f6a1ae18f0c4 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:47048
I20240503 02:13:16.341642 31695 ts_tablet_manager.cc:1875] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.347468 31695 ts_tablet_manager.cc:1888] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240503 02:13:16.347591 31695 log.cc:1196] T c81f82e2608a401ba8f4f381c3ad95be P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/c81f82e2608a401ba8f4f381c3ad95be
I20240503 02:13:16.348266 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet c81f82e2608a401ba8f4f381c3ad95be (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:16.348278 31695 tablet_replica.cc:330] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:16.348493 31695 raft_consensus.cc:2229] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.348685 31695 raft_consensus.cc:2258] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.349411 31695 ts_tablet_manager.cc:1875] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.353874 31695 ts_tablet_manager.cc:1888] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 02:13:16.353981 31695 log.cc:1196] T 441b8fdd54574e0f837d04e0916d1bf1 P d7b4ee58dad047fe8c54f6a1ae18f0c4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-2-root/wals/441b8fdd54574e0f837d04e0916d1bf1
I20240503 02:13:16.354593 30595 catalog_manager.cc:4830] TS d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): tablet 441b8fdd54574e0f837d04e0916d1bf1 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:16.357055 31640 raft_consensus.cc:1050] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: attempting to promote NON_VOTER 4c1727d9448f4a9c81e2c532e5339801 to VOTER
I20240503 02:13:16.357729 31640 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:16.359961 31107 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEARNER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.360050 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.360680 30707 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.360682 31673 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.361095 31640 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.361862 31661 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.365146 31673 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.365655 30707 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.369947 30610 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 reported cstate change: config changed from index 32 to 33, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.370921 31256 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.376544 31106 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 4c1727d9448f4a9c81e2c532e5339801 (127.26.178.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.378495 30787 consensus_queue.cc:237] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } }
I20240503 02:13:16.380553 31256 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.381265 31673 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:16.381848 31107 raft_consensus.cc:1261] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer 7a9cbb0303f446618b0e8bc7ad35afb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.382283 31673 consensus_queue.cc:1035] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:16.383986 31661 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.384739 31107 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.386771 30596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8c3012068484fe3acd108cc36eb62d5 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.388581 31256 raft_consensus.cc:2932] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.388654 30610 catalog_manager.cc:5484] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 33 to 34, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 2 leader_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } }
I20240503 02:13:16.392172 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c3012068484fe3acd108cc36eb62d5 with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:16.392863 31701 tablet_replica.cc:330] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:16.393131 31701 raft_consensus.cc:2229] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.394510 31701 raft_consensus.cc:2258] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.395736 31701 ts_tablet_manager.cc:1875] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.402511 31701 ts_tablet_manager.cc:1888] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240503 02:13:16.402745 31701 log.cc:1196] T c8c3012068484fe3acd108cc36eb62d5 P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/c8c3012068484fe3acd108cc36eb62d5
I20240503 02:13:16.403451 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet c8c3012068484fe3acd108cc36eb62d5 (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:16.415654 31702 raft_consensus.cc:1050] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:16.416568 31702 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 32, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:16.418694 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 02:13:16.418879 31256 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 02:13:16.419477 30707 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 02:13:16.419494 31677 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 02:13:16.419978 31702 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 02:13:16.420372 31704 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 02:13:16.423194 31677 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.423715 31256 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.426875 31182 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.427858 30606 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: config changed from index 34 to 35, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.432922 30855 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:16.432888 30707 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } attrs { replace: true } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.435004 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240503 02:13:16.435559 31552 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 02:13:16.436738 31256 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240503 02:13:16.437981 31702 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 02:13:16.437815 31552 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.439204 31256 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.441833 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e3d6429d7dd499fbe2957fb46fdf30b with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.441773 31181 raft_consensus.cc:2932] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.446063 30605 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 35 to 36, VOTER ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65) evicted. New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:16.449800 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3d6429d7dd499fbe2957fb46fdf30b with delete_type TABLET_DATA_TOMBSTONED (TS ebd1bb9e6d2f4384b781596ce1795b54 not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:60488
I20240503 02:13:16.450170 31701 tablet_replica.cc:330] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:16.450347 31701 raft_consensus.cc:2229] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.450605 31701 raft_consensus.cc:2258] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.452070 31701 ts_tablet_manager.cc:1875] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.457692 31701 ts_tablet_manager.cc:1888] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.35
I20240503 02:13:16.457830 31701 log.cc:1196] T 4e3d6429d7dd499fbe2957fb46fdf30b P ebd1bb9e6d2f4384b781596ce1795b54: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-0-root/wals/4e3d6429d7dd499fbe2957fb46fdf30b
I20240503 02:13:16.458503 30597 catalog_manager.cc:4830] TS ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): tablet 4e3d6429d7dd499fbe2957fb46fdf30b (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:16.544584 31677 raft_consensus.cc:1050] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER 3e8a84e16aa2470899683bc9edef9283 to VOTER
I20240503 02:13:16.545260 31677 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:16.547380 31107 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.547456 31256 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.547466 30784 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 02:13:16.547977 31552 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.548274 31677 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.548542 31704 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.552027 31704 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.552587 31107 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.552842 31256 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.556329 30606 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 32 to 33, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:16.558449 30784 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.560940 30855 consensus_queue.cc:237] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:16.563081 31107 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.563756 31704 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
I20240503 02:13:16.563457 31256 raft_consensus.cc:1261] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 02:13:16.566272 31702 consensus_queue.cc:1035] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:13:16.567174 30784 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 02:13:16.567648 31702 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:16.568224 31677 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.568759 31107 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.571624 31181 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEARNER]: Advancing to term 4
I20240503 02:13:16.571664 30707 raft_consensus.cc:3037] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:13:16.573920 30707 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 02:13:16.575210 31702 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:13:16.575744 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60cc6188939e4b728b1c1efea836a7bb with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.575752 31255 raft_consensus.cc:2932] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.577111 30606 catalog_manager.cc:5484] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 reported cstate change: config changed from index 33 to 34, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 2 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "4c1727d9448f4a9c81e2c532e5339801" member_type: VOTER last_known_addr { host: "127.26.178.68" port: 40457 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } } }
I20240503 02:13:16.578665 31181 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 02:13:16.579281 31552 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: NON_VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:13:16.583870 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 60cc6188939e4b728b1c1efea836a7bb with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:16.584648 31717 tablet_replica.cc:330] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:16.585110 31717 raft_consensus.cc:2229] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.585408 31717 raft_consensus.cc:2258] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.586741 31717 ts_tablet_manager.cc:1875] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.595748 31717 ts_tablet_manager.cc:1888] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240503 02:13:16.596536 31717 log.cc:1196] T 60cc6188939e4b728b1c1efea836a7bb P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/60cc6188939e4b728b1c1efea836a7bb
I20240503 02:13:16.596844 31677 raft_consensus.cc:1050] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: attempting to promote NON_VOTER cf667d67d5b341c7af3b984e0c1f2d55 to VOTER
I20240503 02:13:16.597487 31677 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:16.599411 31182 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 LEARNER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 02:13:16.599592 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet 60cc6188939e4b728b1c1efea836a7bb (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:16.599779 30707 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 02:13:16.600144 31714 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:16.600402 31677 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:16.600947 30784 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 02:13:16.601374 31714 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:13:16.603180 31677 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 29 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.603752 31182 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.604300 30784 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.608431 30610 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 reported cstate change: config changed from index 29 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.608572 30707 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "7a9cbb0303f446618b0e8bc7ad35afb4" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 46225 } attrs { replace: true } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.613858 30861 consensus_queue.cc:237] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } }
I20240503 02:13:16.615584 30707 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 02:13:16.616137 31713 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.616704 31182 raft_consensus.cc:1261] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Refusing update from remote peer d7b4ee58dad047fe8c54f6a1ae18f0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 02:13:16.617338 31713 consensus_queue.cc:1035] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:13:16.618695 31677 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.621722 30595 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 106b5308c1b34e119da27c3c75178ecb with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:16.622403 30707 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.622738 31182 raft_consensus.cc:2932] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } }
I20240503 02:13:16.623040 30609 catalog_manager.cc:5484] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 reported cstate change: config changed from index 32 to 33, VOTER 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66) evicted. New cstate: current_term: 4 leader_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ebd1bb9e6d2f4384b781596ce1795b54" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:16.626994 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 106b5308c1b34e119da27c3c75178ecb with delete_type TABLET_DATA_TOMBSTONED (TS 7a9cbb0303f446618b0e8bc7ad35afb4 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:42326
I20240503 02:13:16.627336 31717 tablet_replica.cc:330] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:16.627544 31717 raft_consensus.cc:2229] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.627806 31717 raft_consensus.cc:2258] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.628620 31717 ts_tablet_manager.cc:1875] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.633145 31717 ts_tablet_manager.cc:1888] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20240503 02:13:16.633275 31717 log.cc:1196] T 106b5308c1b34e119da27c3c75178ecb P 7a9cbb0303f446618b0e8bc7ad35afb4: Deleting WAL directory at /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714702282772525-27337-0/minicluster-data/ts-1-root/wals/106b5308c1b34e119da27c3c75178ecb
I20240503 02:13:16.633940 30596 catalog_manager.cc:4830] TS 7a9cbb0303f446618b0e8bc7ad35afb4 (127.26.178.66:46225): tablet 106b5308c1b34e119da27c3c75178ecb (table test-workload [id=87461bef0fd849ac915fd86a51cee96f]) successfully deleted
I20240503 02:13:17.512157 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:17.515291 30861 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
mode: GRACEFUL
new_leader_uuid: "3e8a84e16aa2470899683bc9edef9283"
from {username='slave'} at 127.0.0.1:47090
I20240503 02:13:17.515576 30861 raft_consensus.cc:595] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Received request to transfer leadership to TS 3e8a84e16aa2470899683bc9edef9283
I20240503 02:13:17.516197 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 02:13:17.516384 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:17.789669 31720 raft_consensus.cc:979] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: : Instructing follower 3e8a84e16aa2470899683bc9edef9283 to start an election
I20240503 02:13:17.789863 31720 raft_consensus.cc:1067] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Signalling peer 3e8a84e16aa2470899683bc9edef9283 to start an election
I20240503 02:13:17.790510 31255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b"
dest_uuid: "3e8a84e16aa2470899683bc9edef9283"
from {username='slave'} at 127.0.0.1:35926
I20240503 02:13:17.790750 31255 raft_consensus.cc:482] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:17.790853 31255 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:13:17.792773 31255 raft_consensus.cc:504] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:17.793303 31255 leader_election.cc:288] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [CANDIDATE]: Term 5 election: Requested vote from peers d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143), cf667d67d5b341c7af3b984e0c1f2d55 (127.26.178.69:43601)
I20240503 02:13:17.793757 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "3e8a84e16aa2470899683bc9edef9283" candidate_term: 5 candidate_status { last_received { term: 4 index: 36 } } ignore_live_leader: true dest_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4"
I20240503 02:13:17.793757 31182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3d6429d7dd499fbe2957fb46fdf30b" candidate_uuid: "3e8a84e16aa2470899683bc9edef9283" candidate_term: 5 candidate_status { last_received { term: 4 index: 36 } } ignore_live_leader: true dest_uuid: "cf667d67d5b341c7af3b984e0c1f2d55"
I20240503 02:13:17.794009 30861 raft_consensus.cc:3032] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:13:17.794029 31182 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:13:17.794121 30861 raft_consensus.cc:729] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7b4ee58dad047fe8c54f6a1ae18f0c4, State: Running, Role: LEADER
I20240503 02:13:17.794389 30861 consensus_queue.cc:260] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 36, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:17.794780 30861 raft_consensus.cc:3037] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:13:17.796877 31182 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8a84e16aa2470899683bc9edef9283 in term 5.
I20240503 02:13:17.797256 31219 leader_election.cc:302] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a84e16aa2470899683bc9edef9283, cf667d67d5b341c7af3b984e0c1f2d55; no voters:
I20240503 02:13:17.797264 30861 raft_consensus.cc:2454] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8a84e16aa2470899683bc9edef9283 in term 5.
I20240503 02:13:17.797816 31721 raft_consensus.cc:2781] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:13:17.798180 31721 raft_consensus.cc:686] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 5 LEADER]: Becoming Leader. State: Replica: 3e8a84e16aa2470899683bc9edef9283, State: Running, Role: LEADER
I20240503 02:13:17.798467 31721 consensus_queue.cc:237] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 36, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } }
I20240503 02:13:17.800741 30605 catalog_manager.cc:5484] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 reported cstate change: term changed from 4 to 5, leader changed from d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67) to 3e8a84e16aa2470899683bc9edef9283 (127.26.178.70). New cstate: current_term: 5 leader_uuid: "3e8a84e16aa2470899683bc9edef9283" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8a84e16aa2470899683bc9edef9283" member_type: VOTER last_known_addr { host: "127.26.178.70" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:18.243023 30861 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240503 02:13:18.243608 31721 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b4ee58dad047fe8c54f6a1ae18f0c4" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 02:13:18.248951 31182 raft_consensus.cc:1261] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 5 FOLLOWER]: Refusing update from remote peer 3e8a84e16aa2470899683bc9edef9283: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240503 02:13:18.249511 31721 consensus_queue.cc:1035] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf667d67d5b341c7af3b984e0c1f2d55" member_type: VOTER last_known_addr { host: "127.26.178.69" port: 43601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 02:13:19.516889 30665 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:19.519250 30665 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:13:19.519531 30665 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:20.271293 27337 tablet_server.cc:178] TabletServer@127.26.178.65:0 shutting down...
W20240503 02:13:20.279299 31222 consensus_peers.cc:485] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 -> Peer ebd1bb9e6d2f4384b781596ce1795b54 (127.26.178.65:40107): Couldn't send request to peer ebd1bb9e6d2f4384b781596ce1795b54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.302815 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.303076 27337 tablet_replica.cc:330] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.303272 27337 raft_consensus.cc:2229] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.303725 27337 raft_consensus.cc:2258] T 441b8fdd54574e0f837d04e0916d1bf1 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.304441 27337 tablet_replica.cc:330] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.304565 27337 raft_consensus.cc:2229] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.304769 27337 raft_consensus.cc:2258] T 5277bc5b18a0494baf73181fb4eeba01 P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.306737 27337 tablet_replica.cc:330] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.307019 27337 raft_consensus.cc:2229] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.307304 27337 raft_consensus.cc:2258] T c81f82e2608a401ba8f4f381c3ad95be P ebd1bb9e6d2f4384b781596ce1795b54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.307963 27337 tablet_replica.cc:330] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.308120 27337 raft_consensus.cc:2229] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.308467 27337 raft_consensus.cc:2258] T d906796832b043e1ae7fe4825491a008 P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.308928 27337 tablet_replica.cc:330] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.309080 27337 raft_consensus.cc:2229] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.309451 27337 raft_consensus.cc:2258] T 4ec68f05310c4dcc8e02dadf0b4934ce P ebd1bb9e6d2f4384b781596ce1795b54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.309984 27337 tablet_replica.cc:330] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54: stopping tablet replica
I20240503 02:13:20.310083 27337 raft_consensus.cc:2229] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.310317 27337 raft_consensus.cc:2258] T 106b5308c1b34e119da27c3c75178ecb P ebd1bb9e6d2f4384b781596ce1795b54 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.327558 27337 tablet_server.cc:194] TabletServer@127.26.178.65:0 shutdown complete.
I20240503 02:13:20.333410 27337 tablet_server.cc:178] TabletServer@127.26.178.66:0 shutting down...
I20240503 02:13:20.349356 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.349716 27337 tablet_replica.cc:330] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.349946 27337 raft_consensus.cc:2229] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.350203 27337 raft_consensus.cc:2258] T 441b8fdd54574e0f837d04e0916d1bf1 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.350942 27337 tablet_replica.cc:330] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.351084 27337 raft_consensus.cc:2229] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.351385 27337 raft_consensus.cc:2258] T c8c3012068484fe3acd108cc36eb62d5 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.351922 27337 tablet_replica.cc:330] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.352051 27337 raft_consensus.cc:2229] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.352272 27337 raft_consensus.cc:2258] T 5277bc5b18a0494baf73181fb4eeba01 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.352708 27337 tablet_replica.cc:330] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.352814 27337 raft_consensus.cc:2229] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.353000 27337 raft_consensus.cc:2258] T 9b2a8cda34b54f4482afed13d9e43c4f P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.353392 27337 tablet_replica.cc:330] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.353495 27337 raft_consensus.cc:2229] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.353600 27337 raft_consensus.cc:2258] T d906796832b043e1ae7fe4825491a008 P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.353979 27337 tablet_replica.cc:330] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4: stopping tablet replica
I20240503 02:13:20.354074 27337 raft_consensus.cc:2229] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.354182 27337 raft_consensus.cc:2258] T 4ec68f05310c4dcc8e02dadf0b4934ce P 7a9cbb0303f446618b0e8bc7ad35afb4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.370865 27337 tablet_server.cc:194] TabletServer@127.26.178.66:0 shutdown complete.
I20240503 02:13:20.392679 27337 tablet_server.cc:178] TabletServer@127.26.178.67:0 shutting down...
I20240503 02:13:20.402402 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.402747 27337 tablet_replica.cc:330] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.402956 27337 raft_consensus.cc:2229] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.403141 27337 raft_consensus.cc:2258] T 3f15a775e24c46eda9ad8f5e8a537da4 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.403776 27337 tablet_replica.cc:330] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.403934 27337 raft_consensus.cc:2229] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.404091 27337 raft_consensus.cc:2258] T 4e3d6429d7dd499fbe2957fb46fdf30b P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.404575 27337 tablet_replica.cc:330] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.404709 27337 raft_consensus.cc:2229] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.404830 27337 raft_consensus.cc:2258] T d906796832b043e1ae7fe4825491a008 P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.405244 27337 tablet_replica.cc:330] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.405377 27337 raft_consensus.cc:2229] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.405642 27337 raft_consensus.cc:2258] T 106b5308c1b34e119da27c3c75178ecb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.406060 27337 tablet_replica.cc:330] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.406185 27337 raft_consensus.cc:2229] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.406365 27337 raft_consensus.cc:2258] T c2e4475d5b86469e90a10f84a910163a P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.406824 27337 tablet_replica.cc:330] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4: stopping tablet replica
I20240503 02:13:20.406952 27337 raft_consensus.cc:2229] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.407194 27337 raft_consensus.cc:2258] T 60cc6188939e4b728b1c1efea836a7bb P d7b4ee58dad047fe8c54f6a1ae18f0c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.423874 27337 tablet_server.cc:194] TabletServer@127.26.178.67:0 shutdown complete.
I20240503 02:13:20.429466 27337 tablet_server.cc:178] TabletServer@127.26.178.68:0 shutting down...
I20240503 02:13:20.438858 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.439128 27337 tablet_replica.cc:330] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.439342 27337 raft_consensus.cc:2229] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.439606 27337 raft_consensus.cc:2258] T 441b8fdd54574e0f837d04e0916d1bf1 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.440248 27337 tablet_replica.cc:330] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.440407 27337 raft_consensus.cc:2229] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.440619 27337 raft_consensus.cc:2258] T c8c3012068484fe3acd108cc36eb62d5 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.441123 27337 tablet_replica.cc:330] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.441256 27337 raft_consensus.cc:2229] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.441428 27337 raft_consensus.cc:2258] T 5277bc5b18a0494baf73181fb4eeba01 P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.441891 27337 tablet_replica.cc:330] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.442018 27337 raft_consensus.cc:2229] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.442265 27337 raft_consensus.cc:2258] T 60cc6188939e4b728b1c1efea836a7bb P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.442744 27337 tablet_replica.cc:330] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.442868 27337 raft_consensus.cc:2229] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.443111 27337 raft_consensus.cc:2258] T 9b2a8cda34b54f4482afed13d9e43c4f P 4c1727d9448f4a9c81e2c532e5339801 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.443516 27337 tablet_replica.cc:330] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801: stopping tablet replica
I20240503 02:13:20.443759 27337 raft_consensus.cc:2229] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.444037 27337 raft_consensus.cc:2258] T c2e4475d5b86469e90a10f84a910163a P 4c1727d9448f4a9c81e2c532e5339801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.458984 27337 tablet_server.cc:194] TabletServer@127.26.178.68:0 shutdown complete.
I20240503 02:13:20.479965 27337 tablet_server.cc:178] TabletServer@127.26.178.69:0 shutting down...
I20240503 02:13:20.489523 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.489786 27337 tablet_replica.cc:330] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.490031 27337 raft_consensus.cc:2229] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.490269 27337 raft_consensus.cc:2258] T 106b5308c1b34e119da27c3c75178ecb P cf667d67d5b341c7af3b984e0c1f2d55 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.490921 27337 tablet_replica.cc:330] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.491032 27337 raft_consensus.cc:2229] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.491139 27337 raft_consensus.cc:2258] T 4e3d6429d7dd499fbe2957fb46fdf30b P cf667d67d5b341c7af3b984e0c1f2d55 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.491566 27337 tablet_replica.cc:330] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.491673 27337 raft_consensus.cc:2229] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.491820 27337 raft_consensus.cc:2258] T c81f82e2608a401ba8f4f381c3ad95be P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.492215 27337 tablet_replica.cc:330] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.492309 27337 raft_consensus.cc:2229] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.492447 27337 raft_consensus.cc:2258] T c2e4475d5b86469e90a10f84a910163a P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.492815 27337 tablet_replica.cc:330] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.492911 27337 raft_consensus.cc:2229] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.493136 27337 raft_consensus.cc:2258] T 3f15a775e24c46eda9ad8f5e8a537da4 P cf667d67d5b341c7af3b984e0c1f2d55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.493497 27337 tablet_replica.cc:330] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55: stopping tablet replica
I20240503 02:13:20.493587 27337 raft_consensus.cc:2229] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.493777 27337 raft_consensus.cc:2258] T 4ec68f05310c4dcc8e02dadf0b4934ce P cf667d67d5b341c7af3b984e0c1f2d55 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 02:13:20.506881 31220 consensus_peers.cc:485] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 -> Peer d7b4ee58dad047fe8c54f6a1ae18f0c4 (127.26.178.67:44143): Couldn't send request to peer d7b4ee58dad047fe8c54f6a1ae18f0c4. Status: Network error: Client connection negotiation failed: client connection to 127.26.178.67:44143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.508265 27337 tablet_server.cc:194] TabletServer@127.26.178.69:0 shutdown complete.
I20240503 02:13:20.512358 27337 tablet_server.cc:178] TabletServer@127.26.178.70:0 shutting down...
I20240503 02:13:20.522003 27337 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:13:20.522282 27337 tablet_replica.cc:330] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.522478 27337 raft_consensus.cc:2229] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.522794 27337 raft_consensus.cc:2258] T 60cc6188939e4b728b1c1efea836a7bb P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.523393 27337 tablet_replica.cc:330] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.523542 27337 raft_consensus.cc:2229] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.523744 27337 raft_consensus.cc:2258] T 3f15a775e24c46eda9ad8f5e8a537da4 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.524217 27337 tablet_replica.cc:330] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.524335 27337 raft_consensus.cc:2229] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.524516 27337 raft_consensus.cc:2258] T 9b2a8cda34b54f4482afed13d9e43c4f P 3e8a84e16aa2470899683bc9edef9283 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.524969 27337 tablet_replica.cc:330] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.525095 27337 raft_consensus.cc:2229] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.525357 27337 raft_consensus.cc:2258] T c81f82e2608a401ba8f4f381c3ad95be P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.525789 27337 tablet_replica.cc:330] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.525923 27337 raft_consensus.cc:2229] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 5 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.526139 27337 raft_consensus.cc:2258] T 4e3d6429d7dd499fbe2957fb46fdf30b P 3e8a84e16aa2470899683bc9edef9283 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.526554 27337 tablet_replica.cc:330] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283: stopping tablet replica
I20240503 02:13:20.526715 27337 raft_consensus.cc:2229] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.526973 27337 raft_consensus.cc:2258] T c8c3012068484fe3acd108cc36eb62d5 P 3e8a84e16aa2470899683bc9edef9283 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.542788 27337 tablet_server.cc:194] TabletServer@127.26.178.70:0 shutdown complete.
I20240503 02:13:20.547104 27337 master.cc:537] Master@127.26.178.126:43777 shutting down...
I20240503 02:13:20.554877 27337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.555104 27337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.555217 27337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 71239a2f32f14ce3921187b9bbcdf86f: stopping tablet replica
I20240503 02:13:20.582427 27337 master.cc:552] Master@127.26.178.126:43777 shutdown complete.
[ OK ] AutoRebalancerTest.TestMaxMovesPerServer (21243 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240503 02:13:20.605084 27337 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.26.178.126:38199
I20240503 02:13:20.605636 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:20.609277 31737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.610077 31738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.610993 31740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:20.611864 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:20.612113 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:20.612169 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:20.612212 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702400612208 us; error 0 us; skew 500 ppm
I20240503 02:13:20.612365 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:20.619274 27337 webserver.cc:460] Webserver started at http://127.26.178.126:46667/ using document root <none> and password file <none>
I20240503 02:13:20.619520 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:20.619596 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:20.619731 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:20.620415 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/master-0-root/instance:
uuid: "91f277a9b96b49609e058d9e5e8bfd37"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.623056 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:20.624640 31745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.624971 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:20.625070 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/master-0-root
uuid: "91f277a9b96b49609e058d9e5e8bfd37"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.625180 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:20.644688 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:20.645215 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:20.658008 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.126:38199
I20240503 02:13:20.658027 31796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.126:38199 every 8 connection(s)
I20240503 02:13:20.659339 31797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.663650 31797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37: Bootstrap starting.
I20240503 02:13:20.664912 31797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.666656 31797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37: No bootstrap required, opened a new log
I20240503 02:13:20.667292 31797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER }
I20240503 02:13:20.667424 31797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.667459 31797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f277a9b96b49609e058d9e5e8bfd37, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.667608 31797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER }
I20240503 02:13:20.667712 31797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:13:20.667752 31797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:13:20.667801 31797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.669570 31797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER }
I20240503 02:13:20.669734 31797 leader_election.cc:302] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f277a9b96b49609e058d9e5e8bfd37; no voters:
I20240503 02:13:20.670068 31797 leader_election.cc:288] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:13:20.670200 31800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.670730 31800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [term 1 LEADER]: Becoming Leader. State: Replica: 91f277a9b96b49609e058d9e5e8bfd37, State: Running, Role: LEADER
I20240503 02:13:20.670879 31797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:20.670943 31800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER }
I20240503 02:13:20.672089 31802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f277a9b96b49609e058d9e5e8bfd37. Latest consensus state: current_term: 1 leader_uuid: "91f277a9b96b49609e058d9e5e8bfd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER } }
I20240503 02:13:20.672065 31801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f277a9b96b49609e058d9e5e8bfd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f277a9b96b49609e058d9e5e8bfd37" member_type: VOTER } }
I20240503 02:13:20.672526 31802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:20.672571 31801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:20.673161 31807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:20.675434 31807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:20.677937 27337 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:13:20.678893 31807 catalog_manager.cc:1347] Generated new cluster ID: 2bb6bbff28f54981acfba20c3120561f
I20240503 02:13:20.678963 31807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:20.690418 31807 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:13:20.691252 31807 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:20.703580 31807 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 91f277a9b96b49609e058d9e5e8bfd37: Generated new TSK 0
I20240503 02:13:20.703902 31807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:20.711042 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:20.714589 31818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.715355 31819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.716291 31821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:20.717118 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:20.717356 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:20.717435 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:20.717459 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702400717456 us; error 0 us; skew 500 ppm
I20240503 02:13:20.717571 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:20.718914 27337 webserver.cc:460] Webserver started at http://127.26.178.65:41299/ using document root <none> and password file <none>
I20240503 02:13:20.719110 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:20.719169 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:20.719261 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:20.719929 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-0-root/instance:
uuid: "fa46a6b598a241c693839f49122774c0"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.722508 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:13:20.724146 31826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.724515 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:20.724608 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-0-root
uuid: "fa46a6b598a241c693839f49122774c0"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.724721 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:20.748950 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:20.749496 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:20.750041 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:20.751039 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:20.751102 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.751183 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:20.751219 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.765547 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.65:38949
I20240503 02:13:20.765568 31888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.65:38949 every 8 connection(s)
I20240503 02:13:20.767280 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:20.771494 31893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.772511 31894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.773622 31896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:20.774747 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:20.775241 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:20.775322 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:20.775359 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702400775355 us; error 0 us; skew 500 ppm
I20240503 02:13:20.775496 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:20.776875 31889 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38199
I20240503 02:13:20.777006 31889 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.777444 31889 heartbeater.cc:511] Master 127.26.178.126:38199 requested a full tablet report, sending...
I20240503 02:13:20.777460 27337 webserver.cc:460] Webserver started at http://127.26.178.66:46153/ using document root <none> and password file <none>
I20240503 02:13:20.777704 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:20.777781 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:20.777916 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:20.778359 31762 ts_manager.cc:194] Registered new tserver with Master: fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949)
I20240503 02:13:20.778613 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-1-root/instance:
uuid: "32aaa4d9e8af497b85142f516dbf23b1"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.779564 31762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59644
I20240503 02:13:20.781512 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:20.783370 31901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.783735 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:20.783843 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-1-root
uuid: "32aaa4d9e8af497b85142f516dbf23b1"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.783952 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:20.802364 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:20.802927 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:20.803539 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:20.804379 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:20.804437 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.804512 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:20.804549 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.819789 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.66:44715
I20240503 02:13:20.819823 31963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.66:44715 every 8 connection(s)
I20240503 02:13:20.821444 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:20.825335 31967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.826005 31968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:20.826951 31970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:20.827739 27337 server_base.cc:1034] running on GCE node
I20240503 02:13:20.827986 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:20.828058 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:20.828092 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702400828089 us; error 0 us; skew 500 ppm
I20240503 02:13:20.828100 31964 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38199
I20240503 02:13:20.828233 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:20.828251 31964 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.828604 31964 heartbeater.cc:511] Master 127.26.178.126:38199 requested a full tablet report, sending...
I20240503 02:13:20.829595 31762 ts_manager.cc:194] Registered new tserver with Master: 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715)
I20240503 02:13:20.829708 27337 webserver.cc:460] Webserver started at http://127.26.178.67:35507/ using document root <none> and password file <none>
I20240503 02:13:20.829916 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:20.829984 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:20.830098 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:20.830441 31762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59646
I20240503 02:13:20.830885 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-2-root/instance:
uuid: "77923e6482a74c4da6b90ec367355ce0"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.833391 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:20.835058 31975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.835731 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:13:20.835852 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-2-root
uuid: "77923e6482a74c4da6b90ec367355ce0"
format_stamp: "Formatted at 2024-05-03 02:13:20 on dist-test-slave-0cbc"
I20240503 02:13:20.836067 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:20.842263 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:20.842864 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:20.843447 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:20.844336 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:20.844406 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.844491 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:20.844530 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:20.859424 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.67:41585
I20240503 02:13:20.859442 32037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.67:41585 every 8 connection(s)
I20240503 02:13:20.866068 32038 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38199
I20240503 02:13:20.866225 32038 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:20.866598 32038 heartbeater.cc:511] Master 127.26.178.126:38199 requested a full tablet report, sending...
I20240503 02:13:20.867424 31762 ts_manager.cc:194] Registered new tserver with Master: 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:20.868135 31762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59654
I20240503 02:13:20.868288 27337 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00792907s
I20240503 02:13:21.780931 31889 heartbeater.cc:503] Master 127.26.178.126:38199 was elected leader, sending a full tablet report...
I20240503 02:13:21.831765 31964 heartbeater.cc:503] Master 127.26.178.126:38199 was elected leader, sending a full tablet report...
I20240503 02:13:21.869547 32038 heartbeater.cc:503] Master 127.26.178.126:38199 was elected leader, sending a full tablet report...
I20240503 02:13:21.896824 27337 test_util.cc:268] Using random seed: 1133478256
I20240503 02:13:21.908834 31762 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59666:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:13:21.909462 31762 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:13:21.943994 31928 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc4f9f1d48a49219ebbea475214910c (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:13:21.944063 31853 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc4f9f1d48a49219ebbea475214910c (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:13:21.944639 31928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc4f9f1d48a49219ebbea475214910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.944649 31853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc4f9f1d48a49219ebbea475214910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.944593 31852 tablet_service.cc:1467] Processing CreateTablet for tablet c404d40aae7a4b2d9e5c85630801c168 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:13:21.944983 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c404d40aae7a4b2d9e5c85630801c168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.944063 31854 tablet_service.cc:1467] Processing CreateTablet for tablet 9488ad74c7514048abad163b1475d260 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:13:21.945469 31854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9488ad74c7514048abad163b1475d260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.945698 32002 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc4f9f1d48a49219ebbea475214910c (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:13:21.946121 32002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc4f9f1d48a49219ebbea475214910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.947053 32003 tablet_service.cc:1467] Processing CreateTablet for tablet 9488ad74c7514048abad163b1475d260 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:13:21.947449 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9488ad74c7514048abad163b1475d260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.950727 32001 tablet_service.cc:1467] Processing CreateTablet for tablet c404d40aae7a4b2d9e5c85630801c168 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:13:21.943994 31929 tablet_service.cc:1467] Processing CreateTablet for tablet 9488ad74c7514048abad163b1475d260 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:13:21.951166 32001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c404d40aae7a4b2d9e5c85630801c168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.951464 31929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9488ad74c7514048abad163b1475d260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.959867 31928 tablet_service.cc:1467] Processing CreateTablet for tablet c404d40aae7a4b2d9e5c85630801c168 (DEFAULT_TABLE table=test-workload [id=3739ddf8bdce44b88b0d62929b44e1a7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:13:21.960530 31928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c404d40aae7a4b2d9e5c85630801c168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.963133 32059 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0: Bootstrap starting.
I20240503 02:13:21.963958 32058 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0: Bootstrap starting.
I20240503 02:13:21.965788 32058 tablet_bootstrap.cc:654] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.966389 32059 tablet_bootstrap.cc:654] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.966949 32060 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1: Bootstrap starting.
I20240503 02:13:21.968364 32059 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0: No bootstrap required, opened a new log
I20240503 02:13:21.968364 32058 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0: No bootstrap required, opened a new log
I20240503 02:13:21.968544 32058 ts_tablet_manager.cc:1367] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:21.968544 32059 ts_tablet_manager.cc:1367] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:13:21.968664 32060 tablet_bootstrap.cc:654] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.969478 32058 raft_consensus.cc:348] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.969478 32059 raft_consensus.cc:348] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.969722 32058 raft_consensus.cc:374] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.969790 32058 raft_consensus.cc:729] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa46a6b598a241c693839f49122774c0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.969731 32059 raft_consensus.cc:374] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.969902 32059 raft_consensus.cc:729] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77923e6482a74c4da6b90ec367355ce0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.970039 32058 consensus_queue.cc:260] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.970099 32059 consensus_queue.cc:260] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.970506 32060 tablet_bootstrap.cc:492] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1: No bootstrap required, opened a new log
I20240503 02:13:21.970744 32060 ts_tablet_manager.cc:1367] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:21.971028 32059 ts_tablet_manager.cc:1398] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:21.971575 32058 ts_tablet_manager.cc:1398] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:21.971710 32059 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0: Bootstrap starting.
I20240503 02:13:21.971940 32058 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: Bootstrap starting.
I20240503 02:13:21.972084 32060 raft_consensus.cc:348] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.972313 32060 raft_consensus.cc:374] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.972435 32060 raft_consensus.cc:729] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aaa4d9e8af497b85142f516dbf23b1, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.972781 32060 consensus_queue.cc:260] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.973527 32059 tablet_bootstrap.cc:654] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.973654 32058 tablet_bootstrap.cc:654] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.981801 32060 ts_tablet_manager.cc:1398] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20240503 02:13:21.982133 32060 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1: Bootstrap starting.
I20240503 02:13:21.982213 32058 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: No bootstrap required, opened a new log
I20240503 02:13:21.982388 32058 ts_tablet_manager.cc:1367] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 02:13:21.983094 32059 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0: No bootstrap required, opened a new log
I20240503 02:13:21.983547 32059 ts_tablet_manager.cc:1367] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:13:21.983325 32058 raft_consensus.cc:348] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.983829 32058 raft_consensus.cc:374] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.983951 32058 raft_consensus.cc:729] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa46a6b598a241c693839f49122774c0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.984400 32060 tablet_bootstrap.cc:654] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.984656 32059 raft_consensus.cc:348] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.984959 32059 raft_consensus.cc:374] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.985041 32059 raft_consensus.cc:729] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77923e6482a74c4da6b90ec367355ce0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.984232 32058 consensus_queue.cc:260] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.985296 32059 consensus_queue.cc:260] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.985617 32058 ts_tablet_manager.cc:1398] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:21.985881 32058 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0: Bootstrap starting.
I20240503 02:13:21.986063 32059 ts_tablet_manager.cc:1398] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:21.986354 32059 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0: Bootstrap starting.
I20240503 02:13:21.987643 32058 tablet_bootstrap.cc:654] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.988158 32059 tablet_bootstrap.cc:654] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.989491 32065 raft_consensus.cc:482] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.989650 32065 raft_consensus.cc:504] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:21.990800 32065 leader_election.cc:288] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715), fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949)
I20240503 02:13:21.990950 32065 raft_consensus.cc:482] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.991124 32060 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1: No bootstrap required, opened a new log
I20240503 02:13:21.991096 32065 raft_consensus.cc:504] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.991286 32060 ts_tablet_manager.cc:1367] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:13:21.991803 32065 leader_election.cc:288] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715)
I20240503 02:13:21.992321 32060 raft_consensus.cc:348] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.992671 32060 raft_consensus.cc:374] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.992769 32060 raft_consensus.cc:729] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aaa4d9e8af497b85142f516dbf23b1, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.993278 32059 tablet_bootstrap.cc:492] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0: No bootstrap required, opened a new log
I20240503 02:13:21.993050 32060 consensus_queue.cc:260] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.993466 32059 ts_tablet_manager.cc:1367] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:13:21.994302 32059 raft_consensus.cc:348] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.994485 32059 raft_consensus.cc:374] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.994531 32059 raft_consensus.cc:729] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77923e6482a74c4da6b90ec367355ce0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.994952 32064 raft_consensus.cc:482] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.994928 32059 consensus_queue.cc:260] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.995117 32064 raft_consensus.cc:504] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.995780 32059 ts_tablet_manager.cc:1398] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:21.997891 32074 raft_consensus.cc:482] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.998096 32058 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0: No bootstrap required, opened a new log
I20240503 02:13:21.998078 32074 raft_consensus.cc:504] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.998320 32058 ts_tablet_manager.cc:1367] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240503 02:13:21.996227 32064 leader_election.cc:288] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:21.999382 32058 raft_consensus.cc:348] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:21.999584 32058 raft_consensus.cc:374] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.999711 32058 raft_consensus.cc:729] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa46a6b598a241c693839f49122774c0, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.999992 32058 consensus_queue.cc:260] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.000643 32058 ts_tablet_manager.cc:1398] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:22.001121 32060 ts_tablet_manager.cc:1398] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:13:22.001987 32060 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1: Bootstrap starting.
I20240503 02:13:22.004307 32060 tablet_bootstrap.cc:654] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.009893 32072 raft_consensus.cc:482] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.010062 32072 raft_consensus.cc:504] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.010979 32072 leader_election.cc:288] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715)
I20240503 02:13:22.013015 32060 tablet_bootstrap.cc:492] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1: No bootstrap required, opened a new log
I20240503 02:13:22.013260 32060 ts_tablet_manager.cc:1367] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 02:13:22.014407 32060 raft_consensus.cc:348] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.014822 32060 raft_consensus.cc:374] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.014951 32060 raft_consensus.cc:729] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aaa4d9e8af497b85142f516dbf23b1, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.015255 32060 consensus_queue.cc:260] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.016000 32060 ts_tablet_manager.cc:1398] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20240503 02:13:22.017913 31890 tablet.cc:2369] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:22.018297 32074 leader_election.cc:288] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:22.021234 32064 raft_consensus.cc:482] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.021440 32064 raft_consensus.cc:504] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:22.022298 32064 leader_election.cc:288] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:22.031661 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0" is_pre_election: true
I20240503 02:13:22.031904 32013 raft_consensus.cc:2454] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aaa4d9e8af497b85142f516dbf23b1 in term 0.
I20240503 02:13:22.032351 31904 leader_election.cc:302] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aaa4d9e8af497b85142f516dbf23b1, 77923e6482a74c4da6b90ec367355ce0; no voters:
I20240503 02:13:22.032609 32074 raft_consensus.cc:2781] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.032683 32074 raft_consensus.cc:482] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.032783 32074 raft_consensus.cc:3037] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.034899 32074 raft_consensus.cc:504] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.035581 32074 leader_election.cc:288] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 election: Requested vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:22.036123 31864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0" is_pre_election: true
I20240503 02:13:22.036291 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0" is_pre_election: true
I20240503 02:13:22.036377 31864 raft_consensus.cc:2454] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 0.
I20240503 02:13:22.036504 31863 raft_consensus.cc:2454] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 0.
I20240503 02:13:22.036516 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0"
I20240503 02:13:22.036684 32013 raft_consensus.cc:3037] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.036417 31862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0" is_pre_election: true
I20240503 02:13:22.037024 31862 raft_consensus.cc:2454] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 0.
I20240503 02:13:22.038179 31979 leader_election.cc:302] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77923e6482a74c4da6b90ec367355ce0, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.038488 32072 raft_consensus.cc:2781] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.038470 31979 leader_election.cc:302] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77923e6482a74c4da6b90ec367355ce0, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.038583 32072 raft_consensus.cc:482] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.038637 32072 raft_consensus.cc:3037] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.038781 31979 leader_election.cc:302] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77923e6482a74c4da6b90ec367355ce0, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.039136 32065 raft_consensus.cc:2781] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.039217 32065 raft_consensus.cc:482] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.039294 32065 raft_consensus.cc:3037] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.041127 32013 raft_consensus.cc:2454] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aaa4d9e8af497b85142f516dbf23b1 in term 1.
I20240503 02:13:22.041673 32072 raft_consensus.cc:504] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:22.042275 32065 raft_consensus.cc:504] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.042421 32072 leader_election.cc:288] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 election: Requested vote from peers 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715), fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949)
I20240503 02:13:22.042572 32072 raft_consensus.cc:2741] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:13:22.042893 32065 leader_election.cc:288] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 election: Requested vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715)
I20240503 02:13:22.044155 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1" is_pre_election: true
I20240503 02:13:22.044427 31939 raft_consensus.cc:2379] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa46a6b598a241c693839f49122774c0 in current term 1: Already voted for candidate 32aaa4d9e8af497b85142f516dbf23b1 in this term.
I20240503 02:13:22.045040 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1" is_pre_election: true
I20240503 02:13:22.045214 31939 raft_consensus.cc:2454] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa46a6b598a241c693839f49122774c0 in term 0.
I20240503 02:13:22.045933 31828 leader_election.cc:302] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aaa4d9e8af497b85142f516dbf23b1, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.046290 32064 raft_consensus.cc:2781] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.046373 32064 raft_consensus.cc:482] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.046464 32064 raft_consensus.cc:3037] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.046921 31904 leader_election.cc:302] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aaa4d9e8af497b85142f516dbf23b1, 77923e6482a74c4da6b90ec367355ce0; no voters:
I20240503 02:13:22.047271 31862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0"
I20240503 02:13:22.047367 32074 raft_consensus.cc:2781] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.047580 32074 raft_consensus.cc:686] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 LEADER]: Becoming Leader. State: Replica: 32aaa4d9e8af497b85142f516dbf23b1, State: Running, Role: LEADER
I20240503 02:13:22.047899 32074 consensus_queue.cc:237] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.048102 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0"
I20240503 02:13:22.048293 31863 raft_consensus.cc:3037] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.050068 32064 raft_consensus.cc:504] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:22.050587 31862 raft_consensus.cc:2379] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 77923e6482a74c4da6b90ec367355ce0 in current term 1: Already voted for candidate fa46a6b598a241c693839f49122774c0 in this term.
I20240503 02:13:22.051559 31863 raft_consensus.cc:2454] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 1.
I20240503 02:13:22.051869 32064 leader_election.cc:288] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 election: Requested vote from peers 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:22.052824 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1"
I20240503 02:13:22.053066 31939 raft_consensus.cc:3037] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.053563 31979 leader_election.cc:302] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77923e6482a74c4da6b90ec367355ce0, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.053869 32083 raft_consensus.cc:2781] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.053984 32083 raft_consensus.cc:686] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 77923e6482a74c4da6b90ec367355ce0, State: Running, Role: LEADER
I20240503 02:13:22.054247 32083 consensus_queue.cc:237] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.057839 31939 raft_consensus.cc:2454] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa46a6b598a241c693839f49122774c0 in term 1.
I20240503 02:13:22.058657 32066 raft_consensus.cc:482] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.058847 32066 raft_consensus.cc:504] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } }
I20240503 02:13:22.059383 31828 leader_election.cc:302] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aaa4d9e8af497b85142f516dbf23b1, fa46a6b598a241c693839f49122774c0; no voters:
I20240503 02:13:22.059762 32066 leader_election.cc:288] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949), 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67:41585)
I20240503 02:13:22.060106 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0" is_pre_election: true
I20240503 02:13:22.060313 31863 raft_consensus.cc:2454] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aaa4d9e8af497b85142f516dbf23b1 in term 0.
I20240503 02:13:22.060257 31862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0"
I20240503 02:13:22.060472 31862 raft_consensus.cc:3037] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.060546 32013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0" is_pre_election: true
I20240503 02:13:22.060801 32013 raft_consensus.cc:2379] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa46a6b598a241c693839f49122774c0 in current term 1: Already voted for candidate 32aaa4d9e8af497b85142f516dbf23b1 in this term.
I20240503 02:13:22.062150 32012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0" is_pre_election: true
I20240503 02:13:22.062482 32012 raft_consensus.cc:2379] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa46a6b598a241c693839f49122774c0 in current term 1: Already voted for candidate 77923e6482a74c4da6b90ec367355ce0 in this term.
I20240503 02:13:22.063009 32012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "fa46a6b598a241c693839f49122774c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0"
I20240503 02:13:22.063256 32012 raft_consensus.cc:2379] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa46a6b598a241c693839f49122774c0 in current term 1: Already voted for candidate 77923e6482a74c4da6b90ec367355ce0 in this term.
I20240503 02:13:22.063649 31829 leader_election.cc:302] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa46a6b598a241c693839f49122774c0; no voters: 32aaa4d9e8af497b85142f516dbf23b1, 77923e6482a74c4da6b90ec367355ce0
I20240503 02:13:22.064368 32086 raft_consensus.cc:2781] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.065191 31862 raft_consensus.cc:2454] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aaa4d9e8af497b85142f516dbf23b1 in term 1.
I20240503 02:13:22.066496 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46a6b598a241c693839f49122774c0" is_pre_election: true
I20240503 02:13:22.067020 31863 raft_consensus.cc:2379] T c404d40aae7a4b2d9e5c85630801c168 P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32aaa4d9e8af497b85142f516dbf23b1 in current term 1: Already voted for candidate 77923e6482a74c4da6b90ec367355ce0 in this term.
I20240503 02:13:22.067673 32064 raft_consensus.cc:2726] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.069222 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1" is_pre_election: true
I20240503 02:13:22.069365 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc4f9f1d48a49219ebbea475214910c" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1" is_pre_election: true
I20240503 02:13:22.069463 31938 raft_consensus.cc:2454] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 0.
I20240503 02:13:22.069613 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1" is_pre_election: true
I20240503 02:13:22.069635 32012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "32aaa4d9e8af497b85142f516dbf23b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77923e6482a74c4da6b90ec367355ce0" is_pre_election: true
I20240503 02:13:22.069815 31939 raft_consensus.cc:2379] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77923e6482a74c4da6b90ec367355ce0 in current term 1: Already voted for candidate fa46a6b598a241c693839f49122774c0 in this term.
I20240503 02:13:22.069772 31936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9488ad74c7514048abad163b1475d260" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1"
I20240503 02:13:22.070008 31936 raft_consensus.cc:2379] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 77923e6482a74c4da6b90ec367355ce0 in current term 1: Already voted for candidate fa46a6b598a241c693839f49122774c0 in this term.
I20240503 02:13:22.069793 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c404d40aae7a4b2d9e5c85630801c168" candidate_uuid: "77923e6482a74c4da6b90ec367355ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aaa4d9e8af497b85142f516dbf23b1"
I20240503 02:13:22.070228 31904 leader_election.cc:302] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aaa4d9e8af497b85142f516dbf23b1; no voters: 77923e6482a74c4da6b90ec367355ce0, fa46a6b598a241c693839f49122774c0
I20240503 02:13:22.070330 31935 raft_consensus.cc:3037] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.071978 31977 leader_election.cc:302] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77923e6482a74c4da6b90ec367355ce0; no voters: 32aaa4d9e8af497b85142f516dbf23b1, fa46a6b598a241c693839f49122774c0
I20240503 02:13:22.072561 32083 raft_consensus.cc:2726] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.072973 32086 raft_consensus.cc:686] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 1 LEADER]: Becoming Leader. State: Replica: fa46a6b598a241c693839f49122774c0, State: Running, Role: LEADER
I20240503 02:13:22.073278 31935 raft_consensus.cc:2454] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77923e6482a74c4da6b90ec367355ce0 in term 1.
I20240503 02:13:22.073225 32086 consensus_queue.cc:237] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } }
I20240503 02:13:22.073371 31761 catalog_manager.cc:5484] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77923e6482a74c4da6b90ec367355ce0 (127.26.178.67). New cstate: current_term: 1 leader_uuid: "77923e6482a74c4da6b90ec367355ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.074144 31762 catalog_manager.cc:5484] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66). New cstate: current_term: 1 leader_uuid: "32aaa4d9e8af497b85142f516dbf23b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.075863 32066 raft_consensus.cc:2726] T c404d40aae7a4b2d9e5c85630801c168 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.075951 31760 catalog_manager.cc:5484] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa46a6b598a241c693839f49122774c0 (127.26.178.65). New cstate: current_term: 1 leader_uuid: "fa46a6b598a241c693839f49122774c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.083699 27337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:22.087304 32089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:22.088356 32090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:22.089257 27337 server_base.cc:1034] running on GCE node
W20240503 02:13:22.090026 32092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:22.106084 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:13:22.106201 27337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:13:22.106227 27337 hybrid_clock.cc:648] HybridClock initialized: now 1714702402106223 us; error 0 us; skew 500 ppm
I20240503 02:13:22.106362 27337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:22.107856 27337 webserver.cc:460] Webserver started at http://127.26.178.68:38035/ using document root <none> and password file <none>
I20240503 02:13:22.108084 27337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:22.108152 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:22.108285 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:22.108932 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-3-root/instance:
uuid: "7873e39e1acd4a3694cc06757356d01c"
format_stamp: "Formatted at 2024-05-03 02:13:22 on dist-test-slave-0cbc"
I20240503 02:13:22.111657 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:13:22.113231 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:22.113570 27337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:13:22.113682 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-3-root
uuid: "7873e39e1acd4a3694cc06757356d01c"
format_stamp: "Formatted at 2024-05-03 02:13:22 on dist-test-slave-0cbc"
I20240503 02:13:22.113795 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task4CUknz/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714702282772525-27337-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:22.122843 27337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:22.123384 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:22.123947 27337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:22.124879 27337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:22.124938 27337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:22.125044 27337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:22.125092 27337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:22.135457 32066 consensus_queue.cc:1035] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:22.138921 32074 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 02:13:22.139000 32074 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 02:13:22.138887 32065 consensus_queue.cc:1035] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.148751 32065 consensus_queue.cc:1035] T c404d40aae7a4b2d9e5c85630801c168 P 77923e6482a74c4da6b90ec367355ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.151676 32165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.26.178.68:43045 every 8 connection(s)
I20240503 02:13:22.151683 27337 rpc_server.cc:307] RPC server started. Bound to: 127.26.178.68:43045
I20240503 02:13:22.156955 32066 consensus_queue.cc:1035] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.161139 32086 consensus_queue.cc:1035] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.161748 32166 heartbeater.cc:347] Connected to a master server at 127.26.178.126:38199
I20240503 02:13:22.161852 32166 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:22.162241 32166 heartbeater.cc:511] Master 127.26.178.126:38199 requested a full tablet report, sending...
I20240503 02:13:22.162950 31762 ts_manager.cc:194] Registered new tserver with Master: 7873e39e1acd4a3694cc06757356d01c (127.26.178.68:43045)
I20240503 02:13:22.163831 31762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59670
I20240503 02:13:22.187678 32086 consensus_queue.cc:1035] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.677670 31815 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:13:22.678346 31815 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:13:22.678498 31815 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:13:22.684152 31862 consensus_queue.cc:237] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } }
I20240503 02:13:22.686093 31935 raft_consensus.cc:1261] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Refusing update from remote peer fa46a6b598a241c693839f49122774c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.686167 32012 raft_consensus.cc:1261] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Refusing update from remote peer fa46a6b598a241c693839f49122774c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.686652 32064 consensus_queue.cc:1035] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.686966 32086 consensus_queue.cc:1035] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.690044 32012 raft_consensus.cc:2932] T 9488ad74c7514048abad163b1475d260 P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
I20240503 02:13:22.689971 31935 raft_consensus.cc:2932] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
W20240503 02:13:22.694032 31828 consensus_peers.cc:485] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 -> Peer 7873e39e1acd4a3694cc06757356d01c (127.26.178.68:43045): Couldn't send request to peer 7873e39e1acd4a3694cc06757356d01c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9488ad74c7514048abad163b1475d260. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.694514 31762 catalog_manager.cc:5484] T 9488ad74c7514048abad163b1475d260 P 32aaa4d9e8af497b85142f516dbf23b1 reported cstate change: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "fa46a6b598a241c693839f49122774c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
I20240503 02:13:22.773375 32087 raft_consensus.cc:2932] T 9488ad74c7514048abad163b1475d260 P fa46a6b598a241c693839f49122774c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } attrs { replace: true } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
I20240503 02:13:22.785071 31935 consensus_queue.cc:237] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } }
I20240503 02:13:22.787218 31862 raft_consensus.cc:1261] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Refusing update from remote peer 32aaa4d9e8af497b85142f516dbf23b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.787576 32012 raft_consensus.cc:1261] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Refusing update from remote peer 32aaa4d9e8af497b85142f516dbf23b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.787858 32169 consensus_queue.cc:1035] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.788156 32181 consensus_queue.cc:1035] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.790938 31862 raft_consensus.cc:2932] T 5fc4f9f1d48a49219ebbea475214910c P fa46a6b598a241c693839f49122774c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
I20240503 02:13:22.791112 32012 raft_consensus.cc:2932] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
W20240503 02:13:22.793406 31903 consensus_peers.cc:485] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 -> Peer 7873e39e1acd4a3694cc06757356d01c (127.26.178.68:43045): Couldn't send request to peer 7873e39e1acd4a3694cc06757356d01c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fc4f9f1d48a49219ebbea475214910c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.795356 31760 catalog_manager.cc:5484] T 5fc4f9f1d48a49219ebbea475214910c P 77923e6482a74c4da6b90ec367355ce0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New cstate: current_term: 1 leader_uuid: "32aaa4d9e8af497b85142f516dbf23b1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
I20240503 02:13:22.798127 32183 ts_tablet_manager.cc:902] T 9488ad74c7514048abad163b1475d260 P 7873e39e1acd4a3694cc06757356d01c: Initiating tablet copy from peer fa46a6b598a241c693839f49122774c0 (127.26.178.65:38949)
F20240503 02:13:22.798463 32183 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.queue_time_us, tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.queue_time_us) (tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.run_cpu_time_us, tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.run_cpu_time_us) (tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.run_wall_time_us, tablet-download-pool-106b5308c1b34e119da27c3c75178ecb.run_wall_time_us) (tablet-download-pool-33618faa82f242b984cb6aa123115462.queue_time_us, tablet-download-pool-33618faa82f242b984cb6aa123115462.queue_time_us) (tablet-download-pool-33618faa82f242b984cb6aa123115462.run_cpu_time_us, tablet-download-pool-33618faa82f242b984cb6aa123115462.run_cpu_time_us) (tablet-download-pool-33618faa82f242b984cb6aa123115462.run_wall_time_us, tablet-download-pool-33618faa82f242b984cb6aa123115462.run_wall_time_us) (tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.queue_time_us, tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.queue_time_us) (tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.run_cpu_time_us, tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.run_cpu_time_us) (tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.run_wall_time_us, tablet-download-pool-3f15a775e24c46eda9ad8f5e8a537da4.run_wall_time_us) (tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.queue_time_us, tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.queue_time_us) (tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.run_cpu_time_us, tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.run_cpu_time_us) (tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.run_wall_time_us, tablet-download-pool-441b8fdd54574e0f837d04e0916d1bf1.run_wall_time_us) (tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.queue_time_us, tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.queue_time_us) (tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.run_cpu_time_us, tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.run_cpu_time_us) (tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.run_wall_time_us, tablet-download-pool-4e3d6429d7dd499fbe2957fb46fdf30b.run_wall_time_us) (tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.queue_time_us, tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.queue_time_us) (tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.run_cpu_time_us, tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.run_cpu_time_us) (tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.run_wall_time_us, tablet-download-pool-4ec68f05310c4dcc8e02dadf0b4934ce.run_wall_time_us) (tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.queue_time_us, tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.queue_time_us) (tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.run_cpu_time_us, tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.run_cpu_time_us) (tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.run_wall_time_us, tablet-download-pool-5277bc5b18a0494baf73181fb4eeba01.run_wall_time_us) (tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.queue_time_us, tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.queue_time_us) (tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.run_cpu_time_us, tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.run_cpu_time_us) (tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.run_wall_time_us, tablet-download-pool-60cc6188939e4b728b1c1efea836a7bb.run_wall_time_us) (tablet-download-pool-9488ad74c7514048abad163b1475d260.queue_time_us, tablet-download-pool-9488ad74c7514048abad163b1475d260.queue_time_us) (tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.queue_time_us, tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.queue_time_us) (tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.run_cpu_time_us, tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.run_cpu_time_us) (tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.run_wall_time_us, tablet-download-pool-9b2a8cda34b54f4482afed13d9e43c4f.run_wall_time_us) (tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.queue_time_us, tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.queue_time_us) (tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.run_cpu_time_us, tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.run_cpu_time_us) (tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.run_wall_time_us, tablet-download-pool-c2e4475d5b86469e90a10f84a910163a.run_wall_time_us) (tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.queue_time_us, tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.queue_time_us) (tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.run_cpu_time_us, tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.run_cpu_time_us) (tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.run_wall_time_us, tablet-download-pool-c81f82e2608a401ba8f4f381c3ad95be.run_wall_time_us) (tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.queue_time_us, tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.queue_time_us) (tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.run_cpu_time_us, tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.run_cpu_time_us) (tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.run_wall_time_us, tablet-download-pool-c8c3012068484fe3acd108cc36eb62d5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714702402 (unix time) try "date -d @1714702402" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006ac9) received by PID 27337 (TID 0x7fbc3dbf8700) from PID 27337; stack trace: ***
I20240503 02:13:22.807502 32169 raft_consensus.cc:2932] T 5fc4f9f1d48a49219ebbea475214910c P 32aaa4d9e8af497b85142f516dbf23b1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7873e39e1acd4a3694cc06757356d01c (127.26.178.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa46a6b598a241c693839f49122774c0" member_type: VOTER last_known_addr { host: "127.26.178.65" port: 38949 } attrs { replace: true } } peers { permanent_uuid: "32aaa4d9e8af497b85142f516dbf23b1" member_type: VOTER last_known_addr { host: "127.26.178.66" port: 44715 } } peers { permanent_uuid: "77923e6482a74c4da6b90ec367355ce0" member_type: VOTER last_known_addr { host: "127.26.178.67" port: 41585 } } peers { permanent_uuid: "7873e39e1acd4a3694cc06757356d01c" member_type: NON_VOTER last_known_addr { host: "127.26.178.68" port: 43045 } attrs { promote: true } } }
@ 0x7fbc73715980 (unknown) at ??:0
@ 0x7fbc68a62fb7 gsignal at ??:0
@ 0x7fbc68a64921 abort at ??:0
@ 0x7fbc6b517dcd google::LogMessage::Fail() at ??:0
@ 0x7fbc6b51bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbc6b5177cc google::LogMessage::Flush() at ??:0
@ 0x7fbc6b518f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbc6cb94742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fbc6cb60f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fbc6cb5acd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fbc7c0278de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fbc7c03fdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fbc7c16a480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fbc7c1a2a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fbc7c185981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fbc7c18570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fbc8102798d std::function<>::operator()() at ??:0
@ 0x7fbc6cb64b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbc6cb6819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbc6cb68019 std::_Function_handler<>::_M_invoke() at ??:0
I20240503 02:13:22.902300 32201 ts_tablet_manager.cc:902] T 5fc4f9f1d48a49219ebbea475214910c P 7873e39e1acd4a3694cc06757356d01c: Initiating tablet copy from peer 32aaa4d9e8af497b85142f516dbf23b1 (127.26.178.66:44715)
@ 0x7fbc8102798d std::function<>::operator()() at ??:0
@ 0x7fbc6cb4301e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbc7370a6db start_thread at ??:0
@ 0x7fbc68b4571f clone at ??:0